Nutanix 超融合：从架构原理到集群部署与上线运维的完整方案 – Johny_Zhao – 博客园

According to architectural principles → Planning → Physical installation → Foundation deployment → Cluster creation → Prism initialization → High availability/backup/security → Online operation and maintenance → Implementation list Organization, each key step is given an actionable command or path.

For production deployment, please refer to the Nutanix official documentation, the current version of Release Notes and the Foundation/Prism installation wizard. The production environment uses AOS + AHV/ESXi + Prism, and Community Edition is not recommended.

1. Architecture principle: first understand the data path, then planning can be based on it

At its core, Nutanix integrates compute, storage, network and management planes into standard x86 nodesthe three-layer architecture of “server + SAN/NAS + virtualization platform” is replaced by a software-defined approach.

1.1 Component Panorama

1.2 Data path and failover

write path: Guest VM → hypervisor → Stargate of local CVM → Write local oplog (SSD) → Simultaneously copy to the oplog of another node through 10GbE → The background sinks to persistent storage.

read path: Read the local copy first; if there is no local copy, read it from the node holding the copy across the network, and then migrate it to the local SSD after reading.which subsequently becomes a local hit.

Stargate failover: If a Stargate process becomes unresponsive twice in a row within 30 secondsanother CVM will redirect the storage path on the host to another healthy Stargate; to prevent frequent switching, the original Stargate needs to be stable for 30 secondsThen restore the original path. This means that when a node fails, the VM’s I/O will briefly experience path switching, but the VM itself will not be aware of it.

1.3 Direct corollary to the deployment phase

• CVM network must be 10GbE: The write replication and read forwarding between CVMs all use 10GbE, and only the node NCC with 1GbE uplink will report WARN/FAIL.

• Production cluster requires at least 3 nodes: RF=2 requires at least 3 fault domains to tolerate single node failure; EC has higher requirements for the number of nodes (see 2.3).

• 192.168.5.0/24 is an internal reserved network segment: This network segment is used for internal communication between CVM and hypervisor and must be avoided in IP planning.

2. Pre-deployment planning

2.1 Scale and node selection

The production environment should start with at least 3 nodes.to support RF=2 and cluster quorum. The starting configuration can be planned in the following directions:

Compatibility red line: Server model, CPU generation, NIC and SSD model from Nutanix HCL must be used. Mixed vendor nodes do not support cluster establishment.。

2.2 Network plane planning

It is recommended to implement physical or VLAN isolation to prevent storage traffic and business traffic from affecting each other.

Each node needs to prepare at least: IPMI IP, Hypervisor management IP, CVM IP, cluster VIP, DNS, NTP, gateway, and VLAN ID. Must avoid 192.168.5.0/24。

2.3 RF, EC and capacity planning

• RF=2: Tolerate at least 1 node failure, and can be satisfied with 3 nodes or more.

• RF=3: Tolerate at least 2 node failures, and 6 or more nodes are more reliable.

• EC (erasure coding) : The cluster must have at least 4 nodescan be enabled; block awareness must be maintained at least 4 blocks under RF=2, and at least 6 blocks under RF=3.. Recommended strips are (4,1) or (4,2)AOS dynamic calculation, it is not recommended to manually change the strip size; the larger the strip, the greater the capacity savings, but the higher the reconstruction cost.. It is strongly recommended that the number of cluster nodes is at least 1 more than “data block + parity block” so that there is room to rebuild the stripe in the event of node failure.。

• Expansion reservation: Reserve 20%–30% when planning Expansion space to avoid capacity shortage after going online.

• Using Nutanix Sizer Estimate vCPU, memory, IOPS, and available capacity.

2.4 Software version and life cycle

Starting from AOS 7.0 and Prism Central pc.2024.3, Nutanix has moved from LTS/STS/eSTS model to unified NCI Release Model: Each upgrade series provides 15 months of full maintenance + an additional 9 months of troubleshooting supporta unified version is released twice a year, and the EOM/EOSL date alignment of AOS, AHV, Prism, NCC, Foundation and other components is. When selecting, the upgrade window should be planned according to this model, rather than following the old LTS rhythm.

3. Physical installation and basic environment

1. Shelving and wiring: Connect the power supply, IPMI, service network and storage network.

2. BIOS/UEFI: Enable VT-x/AMD-V, SR-IOV (if necessary), NUMA, and boot using UEFI.

3. IPMI configuration:Set the out-of-band IP, host name, and out-of-band account.

4. Switch side：

   • The storage network port opens the corresponding VLAN;

   • Configure access/trunk for business network ports as needed;

   • If LACP is used, the AHV/ESXi side and switch side must be configured at the same timeunilateral configuration will cause link abnormalities.

5. NTP preflight: Confirm that the time sources of all nodes are consistent and NTP is reachable.

4. Use Foundation to deploy nodes

Foundation’s working mode is: run the GUI in a management VM (Foundation VM), boot the target node through network PXE, automatically complete the installation of the Hypervisor and AOS CVM, and create the initial cluster.

4.1 Prepare Foundation environment

1. Import the OVF/OVA of Foundation VM in vSphere or AHV, and allocate an IP reachable by the management network.

2. Download the AOS installation package, Hypervisor ISO, and Foundation version matching package that match the current version。

3. Double-click Nutanix Foundation on the Foundation VM desktopthe browser opens http://localhost:8000/gui/index.htmlcan also be accessed from outside http://:8000/gui/index.html。

4.2 Import node configuration

Recommended method:Log in install.nutanix.com Create a Node Configuration file, fill in the model, CVM IP, Hypervisor IP, IPMI IP, cluster VIP, time zone, DNS, NTP, etc. of each node. After exporting, import the configuration file on the Start page of the Foundation GUI.. Then upload the AOS and Hypervisor images separately in the GUI(Configuration file does not contain images).

If you choose to fill it in manually, after scanning the nodes in Foundation, confirm the IPMI, serial number, disk, and network card information one by one, and configure the IP, gateway, DNS, NTP, cache disk, capacity disk, and Hypervisor type of IPMI/Hypervisor/CVM for each node.

4.3 Execute Imaging and Cluster Creation

1. Confirm on the Nodes page that all target nodes have been discovered.

2. SelectCreate Clustercheck the node list, CVM IP, and cluster VIP.

3. Start Imaging. Do not interrupt the network or power outage during the process。

4. Once complete, each node should be accessible via: AHV root Login, CVM nutanix Login, Prism Element https://:9440. The default credentials are based on the current version of the document and must be modified immediately after the installation is completed.。

If the node is already running DiscoveryOS but AOS/Hypervisor has not been installed, mirroring must be completed through Foundation first. Add nodes to an existing clusterWhen doing this, do not run Foundation a second time to create a new cluster. Instead, use Expand Cluster in Prism.。

5. Create a Nutanix cluster (manual path when Foundation is not automatically created)

If Foundation does not automatically complete the cluster creation, you can manually execute it on the CVM of any node:

Single node CE environment use --redundancy_factor=1but it is not recommended for production environments. After the manual creation is completed, you still need to enter Prism to complete the initial configuration in Section 6.

6. Prism Element initialization configuration

6.1 First task: running NCC

Log in Afterwards, the first thing to do is not to allocate business, but to run a health check. On any CVM with nutanix User executes:

ncc health_checks run_all

You can also use Health → Actions → Run Checks in the Prism Web Console. implement. Do not continue to configure or launch services before any non-PASS items are resolved.. Key inspections include 10gbe_check(Whether CVM is bound to 10GbE uplink), certificate consistency, NTP synchronization, DNS reachability, disk usage, etc.. If the cluster load is high resulting in run_all timeout, can be used instead ncc --health_checks Execute key items individually。

6.2 Time zone, NTP, password, SMTP

ncli
ncli> cluster set-timezone timezone=Asia/Shanghai

After modifying the time zone, all CVMs need to be restarted on a rolling basis.: Only restart one at a time, and confirm that the CVM is fully restored before operating the next one. NTP recommends pointing to at least 5 stable low stratum sources.

Modify CVM’s nutanix User password and AHV root Password; NCC will check the default password, modify it and re-run NCC to confirm that the alarm disappears. Configure SMTP in Prism so that NCC results and alarms can be delivered by email; enable Pulse/remote support as needed.

6.3 Storage Pool and Storage Container

Foundation will automatically create a default storage pool and a container, but the production environment should be adjusted according to the business plan: confirm the RF settings of the container, whether compression/deduplication is enabled, and whether EC is enabled for a specific container. Before enabling EC, confirm that the number of cluster nodes meets the constraints of 2.3.

6.4 Virtual machine network

In Prism: Gear icon → Network Configuration → Create Networkfill in the VLAN ID and subnet. Also available on CVM via aCLI:

Verify bridge and uplink:

manage_ovs --bridge_name br0 show_uplinks

Confirm that the uplink is 10GbE and the bond is normal. VMs that require multiple VLANs can create trunked NICs:

acli vm.nic_create > network=> \
  trunked_networks=2> vlan_mode=kTrunked

6.5 Image upload and test VM

Upload ISO or VM image, create a test VM, verify network connectivity, storage read and write, snapshot and migration. This is the minimum closed-loop verification before going online, don’t skip it.

7. Prism Central deployment (centralized management)

Prism Element manages a single cluster; Prism Central provides multi-cluster management, Categories, Calm, Flow and capacity planning. Production environment recommendation 3-VM Prism Central To implement HA, 1-VM can be used in small environments.

Deployment method (1-Click):

1. Home in Prism Element Find the Prism Central widget on the page and click Register or create new。

2. SelectDeployspecify the specifications (1-VM or 3-VM), storage container and network.

3. Fill in the IP, gateway, DNS, and NTP of Prism Central.

4. ClickDeploymonitor the status to OK on the Tasks page。

5. Register the Prism Element cluster with Prism Central.

Specification reference(After pc.2023.4.0.2, the memory per PC VM has been increased by 2 GB)：

If you plan to enable optional functions such as Flow, Calm/Self-Service, and DRaaS, you need to add additional CPU/memory resources on this basis.。

8. High availability, data protection and security

8.1 High availability

• Enable VM HA: After a node failure, the VM is restarted on other nodes.

• Anti-Affinity: Key VMs are configured with anti-affinity to avoid being concentrated on the same node.

• Troubleshooting: Test node power outage, disk failure, and CVM restart at least onceconfirm that VM behavior and recovery times are as expected.

• Enable Rebuild Capacity Reservation：在 Prism Element 中 Settings → Data Resiliency → Rebuild Capacity Reservationcheck Reserve Rebuild Capacity. When enabled, the cluster will reserve reconstruction capacity based on Failure Domain and Fault Tolerance, and will be automatically rebuilt when a node fails; when not enabled, the cluster may continue to accept writes without fully protecting the data.after enabling, if new writes cannot be fully protected, the cluster will reject writes. Clusters with 3 nodes or more support this function; it cannot be enabled when RF=1; after enabling it, the Failure Domain cannot be changed at will.。

8.2 Data protection

• Configure snapshot policy和 Protection Domain。

• If disaster recovery is required, configure asynchronous replication或 Metro Availability。

• Backup follows the 3-2-1 principlecombined with third-party backup tools, do not use snapshots as the only backup.

8.3 Security hardening

• Change all default passwords；

• Restrict access to the Prism management network segment;

• Enable data-at-rest encryption；

• Use RBAC least privilege；

• Enable audit logging；

• AHV environments can be accessed via Nutanix Flow Do micro-segmentation.

8.4 AHV memory super production attitude

AHV creates a VM with fixed memory by default. Memory Overcommit needs to be manually enabled on the VM.. Officially it is not recommended to enable it by default in the production environment.: Super dependent on ballooning and swapping. If the VM active memory exceeds the physical memory, continuous paging and performance degradation will occur.. If you do need to enable it for testing or non-critical loads, note: GPU passthrough and vNUMA are not supported on VMs;Cannot hot add memory after enabled;Cross-Cluster Live Migration cannot be used; Windows VM needs to install VirtIO driver package or NGT to have balloon driver. A reasonable approach for production environments is to use AHV’s VirtIO ballooning without exceeding the premise.Increase VM density instead of relying on overscores.

9. Operation and maintenance rhythm after going online

9.1 Daily inspection

• Prism alerts and NCC key results;

• All CVM status (ncc health_checks run_all or Prism Health);

• Storage pool capacity and Rebuild Capacity Reservation margin；

• Data reconstruction progress (if there is a node/disk failure).

9.2 Weekly inspection

9.3 Monthly change window

• AOS/AHV/firmware upgradegive priority to verification in the test environment, and plan according to the maintenance window of the NCI Release Model;

• Using LCM (Life Cycle Manager) Unified management of upgrades to avoid manual mixing of versions;

• Before upgrading, confirm that all NCC PASS and Rebuild Capacity Reservation have sufficient margin.

9.4 Capacity and fault management

• 3–6 months in advanceAssess expansion needs, combining Sizer and actual usage;

• Regularly handle node failures, disk failures, VM migration and recoverydrill;

• Keep configuration baseline: Network, storage, VM templates, backup strategies, upgrade records.

9.5 Expansion

When adding nodes to an existing cluster, use Expand Cluster in Prismdo not run Foundation twice to create a new cluster. The new node must be on the same HCL platform, same model, or compatible model as the existing node.

10. Minimum deployable list

It is recommended to use 3 nodes for POC when building for the first time.: Completely go through Foundation installation → Cluster creation → Prism initialization → Test VM → Snapshot recovery → Node failover → Upgrade drill, and then incorporate it into production business after confirmation of stability.

Attached: Quick check on key pit avoidance methods

1. IP planning avoids 192.168.5.0/24otherwise CVM–Hypervisor internal communication may be abnormal.

2. Do not mix vendor nodes to build a clusterall nodes must be on the same HCL platform.

3. 10GbE is the baselinenot a “recommendation”; 1GbE uplinks alone will cause NCC FAIL and severe degradation in inter-CVM replication performance.

4. After modifying the time zone, CVM must be restarted on a rolling basisone at a time, confirm recovery before continuing.

5. AHV Memory Overcommit is not enabled by default in the production environment; If you need to use it, fully evaluate the balloon driver, swap and performance impact.

6. Use Expand Cluster for capacity expansion, do not run Foundation twice.。

7. EC should not be forcibly enabled when the number of nodes is insufficient.: RF=2 at least 4 nodes, RF=3 at least 6 nodes, strip (4,1)/(4,2) is dynamically calculated by AOS.

8. Rebuild Capacity Reservation is enabled as soon as it goes onlineand pay attention to the reservation margin during daily inspections; when approaching the upper limit, the cluster will reject new writes。