这几天HOMELAB的RAC出了奇怪的问题，RAC21节点的集群服务中ora.crsd与ora.storage始终offline，网上找了一圈也没有修复(排错过程就不扯了，Windows下和Linux很多地方不一样，而且19c的排错文章几乎没有)。只好尝试顺序移除，重新构建RAC环境。
Step 1 移除OEM Agent
这个最简单，直接借助OUI即可，没有任何报错。
Step 2 移除OGG 19
很多文章写的很简单，但实际上不是那么回事儿。通过OUI移除时，OUI界面始终无反应。通过deinstall命令行移除，依然报错：
PS C:\app\19.0.0\ogg> cd .\deinstall\
PS C:\app\19.0.0\ogg\deinstall> .\deinstall.bat
ALERT: Ensure all the processes running from the current Oracle Home are shutdown prior to running this software uninstallation script.
Proceed with removing Oracle GoldenGate home: C:\app\19.0.0\ogg (yes/no)? [no] yes
只好按照Oracle官方建议，ggsci下手工停掉所有进程、移除注册信息、删除服务；ogg安装目录下.\install deleteevents deleteservice删除添加的系统服务项；
最后再强制删除目录，如果被阻止删除，则重启下OS可以顺利删除。
如果后面重新安装OGG到相同目录时会报错INS-75012，此时还需要手工删除注册路径如下：
Step 3 删除RAC21节点
这是按照官方流程操作，先在RAC21上运行以下命令：
https://docs.oracle.com/en/database/oracle/oracle-database/19/cwadd/adding-and-deleting-cluster-nodes.html#GUID-DDED83A3-F1BB-43EB-AFA4-FDAF51E84EAFPS C:\app\19.0.0\grid\oui\bin> .\setup.exe -updateNodeList ORACLE_HOME=Grid_home "CLUSTER_NODES=rac21" CRS=TRUE -local
PS C:\app\19.0.0\grid\oui\bin> cd ..
PS C:\app\19.0.0\grid\oui> cd ..
PS C:\app\19.0.0\grid> cd .\deinstall\
PS C:\app\19.0.0\grid\deinstall> .\deinstall.bat -local
Checking for required files and bootstrapping ...
Please wait ...
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
Location of logs C:\Program Files\Oracle\Inventory\logs\
############ ORACLE DECONFIG TOOL START ############
######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##
Checking for existence of the Oracle home location C:\app\19.0.0\grid
Oracle Home type selected for deinstall is: Oracle Grid Infrastructure for a Cluster
Oracle Base selected for deinstall is: C:\app\root
Checking for existence of central inventory location C:\Program Files\Oracle\Inventory
Checking for existence of the Oracle Grid Infrastructure home C:\app\19.0.0\grid
The following nodes are part of this cluster: rac21
## [END] Install check configuration ##
Traces log file: C:\Program Files\Oracle\Inventory\logs\\crsdc_2020-02-09_02-42-29-PM.log
Network Configuration check config START
Network de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\netdc_check2020-02-09_02-42-43PM.log
Network Configuration check config END
Asm Check Configuration START
ASM de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\asmcadc_check2020-02-09_02-42-43PM.log
Database Check Configuration START
Database de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\databasedc_check2020-02-09_02-42-43PM.log
Oracle Grid Management database was not found in this Grid Infrastructure home
Database Check Configuration END
######################### DECONFIG CHECK OPERATION END #########################
####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is: C:\app\19.0.0\grid
The following nodes are part of this cluster: rac21
The cluster node(s) on which the Oracle home deinstallation will be performed are:rac21
Oracle Home selected for deinstall is: C:\app\19.0.0\grid
Inventory Location where the Oracle home registered is: C:\Program Files\Oracle\Inventory
Option -local will not modify any ASM configuration.
Oracle Grid Management database was not found in this Grid Infrastructure home
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_02-42-27-PM.out'
Any error messages from this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_02-42-27-PM.err'
######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\databasedc_clean2020-02-09_02-42-43PM.log
ASM de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\asmcadc_clean2020-02-09_02-42-43PM.log
ASM Clean Configuration END
Network Configuration clean config START
Network de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\netdc_clean2020-02-09_02-42-43PM.log
Network Configuration clean config END
Root script execution in progress on nodes rac21
## [START] Removing Oracle Home User configuration ##
Removing home specific groups.
Removing group ORA_OraGI19Home1_OPER
Removing group ORA_OraGI19Home1_DBA
Removing group ORA_OraGI19Home1_SYSBACKUP
Removing group ORA_OraGI19Home1_SYSDG
Removing group ORA_OraGI19Home1_SYSKM
## [END] Removing Oracle Home User configuration ##
######################### DECONFIG CLEAN OPERATION END #########################
####################### DECONFIG CLEAN OPERATION SUMMARY #######################
There is no Oracle Grid Management database to de-configure in this Grid Infrastructure home
Oracle Clusterware is stopped and successfully de-configured on node "rac21"
Oracle Clusterware is stopped and de-configured successfully.
#######################################################################
############# ORACLE DECONFIG TOOL END #############
Using properties file C:\Users\root\AppData\Local\Temp\deinstall2020-02-09_02-42-18PM\response\deinstall_2020-02-09_02-42-27-PM.rsp
Location of logs C:\Program Files\Oracle\Inventory\logs\
############ ORACLE DEINSTALL TOOL START ############
####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_02-42-27-PM.out'
Any error messages from this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_02-42-27-PM.err'
######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to rac21
Setting CLUSTER_NODES to rac21
Setting CRS_HOME to true
Setting oracle.installer.local to true
## [END] Preparing for Deinstall ##
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START
Detach Oracle home 'C:\app\19.0.0\grid' from the central inventory on the local node : Done
Failed to delete the directory 'C:\app\19.0.0\grid\perl\bin'. Either user has no permission to delete or it is in use.
Failed to delete the directory 'C:\app\19.0.0\grid\perl'. The directory is not empty.
Failed to delete the directory 'C:\app\19.0.0\grid'. The directory is not empty.
Delete directory 'C:\app\19.0.0\grid' on the local node : Failed <<<<
Delete directory 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Oracle - OraGI19Home1' on the local node : Done
Removing oracle home 'C:\app\19.0.0\grid' from PATH variable on the local node : Done
The Oracle Base directory 'C:\app\root' will not be removed on local node. The directory is in use by Oracle Home 'C:\app\19.0.0\database'.
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END
## [START] Oracle install clean ##
## [END] Oracle install clean ##
######################### DEINSTALL CLEAN OPERATION END #########################
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home 'C:\app\19.0.0\grid' from the central inventory on the local node.
Failed to delete directory 'C:\app\19.0.0\grid' on the local node due to error : Either user has no permission to delete or file is in use.
Review the permissions and manually delete 'C:\app\19.0.0\grid' on local node.
Successfully deleted directory 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Oracle - OraGI19Home1' on the local node.
Successfully removed oracle home 'C:\app\19.0.0\grid' from PATH variable on the local node.
Oracle Universal Installer cleanup was successful.
Review the permissions and contents of 'C:\app\root' on nodes(s) 'rac21'.
If there are no Oracle home(s) associated with 'C:\app\root', manually delete 'C:\app\root' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL TOOL END #############
PS C:\app\19.0.0\grid\deinstall>接下来的命令在RAC22上运行(先移除RAC22节点的DNS配置)：
PS C:\Windows\system32> cd C:\app\19.0.0\grid\bin\
PS C:\app\19.0.0\grid\bin> .\crsctl.exe delete node -n rac21
CRS-4661: Node rac21 successfully deleted.
PS C:\app\19.0.0\grid\bin> .\cluvfy.bat stage -post nodedel -n rac21
PRVF-0002 : could not retrieve local node name
rac22: DNS server failure [response code 2]
PS C:\app\19.0.0\grid\bin> .\cluvfy.bat stage -post nodedel -n rac22
PRVF-0002 : could not retrieve local node name
rac22: DNS server failure [response code 2]
PS C:\app\19.0.0\grid\bin> .\cluvfy.bat stage -post nodedel -n rac22
java.lang.RuntimeException: no nameservers provided
at sun.net.spi.nameservice.dns.DNSNameService.<init>(DNSNameService.java:256)
at sun.net.spi.nameservice.dns.DNSNameServiceDescriptor.createNameService(DNSNameServiceDescriptor.java:35)
at java.net.InetAddress$3.run(InetAddress.java:951)
at java.net.InetAddress$3.run(InetAddress.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.InetAddress.createNSProvider(InetAddress.java:939)
at java.net.InetAddress.<clinit>(InetAddress.java:984)
at sun.management.VMManagementImpl.getVmId(VMManagementImpl.java:140)
at sun.management.RuntimeImpl.getName(RuntimeImpl.java:59)
at oracle.ops.mgmt.trace.Trace.<clinit>(Trace.java:198)
at oracle.ops.verification.client.CluvfyDriver.main(CluvfyDriver.java:224)
Cannot create name service:dns,sun: java.lang.RuntimeException: no nameservers provided
Node removal check failed
The following node is in cluster: rac22
Post-check for node removal was unsuccessful on all the nodes.
CVU operation performed: stage -post nodedel
Date: Feb 9, 2020 2:49:18 PM
CVU home: C:\app\19.0.0\grid\bin\..\
User: CITRIX\root
PS C:\app\19.0.0\grid\bin> .\cluvfy.bat stage -post nodedel -n rac21
java.lang.RuntimeException: no nameservers provided
at sun.net.spi.nameservice.dns.DNSNameService.<init>(DNSNameService.java:256)
at sun.net.spi.nameservice.dns.DNSNameServiceDescriptor.createNameService(DNSNameServiceDescriptor.java:35)
at java.net.InetAddress$3.run(InetAddress.java:951)
at java.net.InetAddress$3.run(InetAddress.java:940)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.InetAddress.createNSProvider(InetAddress.java:939)
at java.net.InetAddress.<clinit>(InetAddress.java:984)
at sun.management.VMManagementImpl.getVmId(VMManagementImpl.java:140)
at sun.management.RuntimeImpl.getName(RuntimeImpl.java:59)
at oracle.ops.mgmt.trace.Trace.<clinit>(Trace.java:198)
at oracle.ops.verification.client.CluvfyDriver.main(CluvfyDriver.java:224)
Cannot create name service:dns,sun: java.lang.RuntimeException: no nameservers provided
Verifying Node Removal ...
Verifying CRS Integrity ...PASSED
Verifying Clusterware Version Consistency ...PASSED
Verifying Node Removal ...PASSED
Post-check for node removal was successful.
CVU operation performed: stage -post nodedel
Date: Feb 9, 2020 2:49:32 PM
CVU home: C:\app\19.0.0\grid\bin\..\
User: CITRIX\root
PS C:\app\19.0.0\grid\bin> .\srvctl.bat config vip -node rac21
VIP exists: network number 1, hosting node rac21
VIP Name: rac21-vip.citrix.org.cn
VIP IPv4 Address: 192.168.0.25
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
PS C:\app\19.0.0\grid\bin> .\srvctl.bat stop vip -node rac21
PRKO-3250 : Failed to stop VIPs on node rac21 because no VIP was running on the given node
PS C:\app\19.0.0\grid\bin> Step 4 移除Database 19c
GRID数据顺利移除，开始移除database，不要用OUI移除，因为OUI一样不起作用：
PS C:\app\19.0.0> cd .\database\
PS C:\app\19.0.0\database> cd .\deinstall\
PS C:\app\19.0.0\database\deinstall> .\deinstall.bat
Checking for required files and bootstrapping ...
Please wait ...
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
Location of logs C:\Program Files\Oracle\Inventory\logs\
############ ORACLE DECONFIG TOOL START ############
######################### DECONFIG CHECK OPERATION START #########################
## [START] Install check configuration ##
Checking for existence of the Oracle home location C:\app\19.0.0\database
Oracle Home type selected for deinstall is: Oracle Real Application Cluster Database
Oracle Base selected for deinstall is: C:\app\root
Checking for existence of central inventory location C:\Program Files\Oracle\Inventory
Checking for existence of the Oracle Grid Infrastructure home
The following nodes are part of this cluster: rac21
## [END] Install check configuration ##
## [START] Checking Windows and .NET products configuration ##
The following Windows and .NET products will be deconfigured from the Oracle home : oledbolap,ode.net,ntoledb,oramts,asp.net,odp.net
## [END] Checking Windows and .NET products configuration ##
Network Configuration check config START
Network de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\netdc_check2020-02-09_03-01-49PM.log
Network Configuration check config END
Database Check Configuration START
Database de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\databasedc_check2020-02-09_03-01-49PM.log
Use comma as separator when specifying list of values as input
Specify the list of database names that are configured in this Oracle home [ORCL1]:
###### For Database 'ORCL1' ######
Specify the type of this database (1.Single Instance Database|2.Oracle Restart Enabled Database|3.RAC Database|4.RAC One Node Database) [1]:
Specify the diagnostic destination location of the database [C:\app\root\diag\rdbms\orcl1]:
Specify the storage type used by the Database ASM|FS []: ASM
Specify if database Archive Mode is Enabled. y/n [n]: y
Database Check Configuration END
######################### DECONFIG CHECK OPERATION END #########################
####################### DECONFIG CHECK OPERATION SUMMARY #######################
Oracle Grid Infrastructure Home is:
The following nodes are part of this cluster: rac21
The cluster node(s) on which the Oracle home deinstallation will be performed are:rac21
Oracle Home selected for deinstall is: C:\app\19.0.0\database
Inventory Location where the Oracle home registered is: C:\Program Files\Oracle\Inventory
The following Windows and .NET products will be deconfigured from the Oracle home : oledbolap,ode.net,ntoledb,oramts,asp.net,odp.net
The following databases were selected for de-configuration. The databases will be deleted and will not be useful upon de-configuration : ORCL1
Database unique name : ORCL1
Storage used : ASM
Do you want to continue (y - yes, n - no)? [n]: y
A log of this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_03-01-48-PM.out'
Any error messages from this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_03-01-48-PM.err'
######################## DECONFIG CLEAN OPERATION START ########################
Database de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\databasedc_clean2020-02-09_03-01-49PM.log
Database Clean Configuration START ORCL1
This operation may take few minutes.
Database Clean Configuration END ORCL1
Network Configuration clean config START
Network de-configuration trace file location: C:\Program Files\Oracle\Inventory\logs\netdc_clean2020-02-09_03-01-49PM.log
De-configuring Naming Methods configuration file...
Naming Methods configuration file de-configured successfully.
De-configuring Local Net Service Names configuration file...
Local Net Service Names configuration file de-configured successfully.
De-configuring backup files...
Backup files de-configured successfully.
The network configuration has been cleaned up successfully.
Network Configuration clean config END
## [START] Removing Windows and .NET products configuration ##
## [END] Removing Windows and .NET products configuration ##
## [START] Removing Oracle Home User configuration ##
Removing home specific groups.
Removing group ORA_OraDB19Home1_OPER
Removing group ORA_OraDB19Home1_DBA
Removing group ORA_OraDB19Home1_SYSBACKUP
Removing group ORA_OraDB19Home1_SYSDG
Removing group ORA_OraDB19Home1_SYSKM
Removing group ORA_OraDB19Home1_SVCSIDS
Removing group ORA_OraDB19Home1_SVCACCTS
## [END] Removing Oracle Home User configuration ##
######################### DECONFIG CLEAN OPERATION END #########################
####################### DECONFIG CLEAN OPERATION SUMMARY #######################
Successfully de-configured the following database instances : ORCL1
Removed oledbolap configuration
Removed ode.net configuration
Removed ntoledb configuration
Removed oramts configuration
Removed asp.net configuration
Removed odp.net configuration
#######################################################################
############# ORACLE DECONFIG TOOL END #############
Using properties file C:\Users\root\AppData\Local\Temp\deinstall2020-02-09_03-01-40PM\response\deinstall_2020-02-09_03-01-48-PM.rsp
Location of logs C:\Program Files\Oracle\Inventory\logs\
############ ORACLE DEINSTALL TOOL START ############
####################### DEINSTALL CHECK OPERATION SUMMARY #######################
A log of this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_03-01-48-PM.out'
Any error messages from this session will be written to: 'C:\Program Files\Oracle\Inventory\logs\deinstall_deconfig2020-02-09_03-01-48-PM.err'
######################## DEINSTALL CLEAN OPERATION START ########################
## [START] Preparing for Deinstall ##
Setting LOCAL_NODE to rac21
Setting CLUSTER_NODES to rac21
Setting CRS_HOME to false
Setting oracle.installer.local to false
## [END] Preparing for Deinstall ##
Setting the force flag to false
Setting the force flag to cleanup the Oracle Base
Oracle Universal Installer clean START
Detach Oracle home 'C:\app\19.0.0\database' from the central inventory on the local node : Done
Delete directory 'C:\app\19.0.0\database' on the local node : Done
Delete directory 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Oracle - OraDB19Home1' on the local node : Done
Removing oracle home 'C:\app\19.0.0\database' from PATH variable on the local node : Done
The Oracle Base directory 'C:\app\root' will not be removed on local node. The directory is not empty. User needs to manually cleanup this directory.
Oracle Universal Installer cleanup was successful.
Oracle Universal Installer clean END
## [START] Oracle install clean ##
## [END] Oracle install clean ##
######################### DEINSTALL CLEAN OPERATION END #########################
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home 'C:\app\19.0.0\database' from the central inventory on the local node.
Successfully deleted directory 'C:\app\19.0.0\database' on the local node.
Successfully deleted directory 'C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Oracle - OraDB19Home1' on the local node.
Successfully removed oracle home 'C:\app\19.0.0\database' from PATH variable on the local node.
Oracle Universal Installer cleanup was successful.
Review the permissions and contents of 'C:\app\root' on nodes(s) 'rac21'.
If there are no Oracle home(s) associated with 'C:\app\root', manually delete 'C:\app\root' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.
#######################################################################
############# ORACLE DEINSTALL TOOL END #############
PS C:\app\19.0.0\database\deinstall>database的移除非常彻底，连安装目录都帮你删除了。移除后再去验证下原有数据库的连接，发现都是正常的。
最后移除注册表中的Oracle父项和C盘相关目录后重启系统开始全新构建RAC的工作。