Over the weekend while I was working on rebuilding my lab, I ran into an issue where a VCF import operation failed because I left an old collector still in the VCF Operations GUI when I was trying to re-import a cluster and ended up costing me some additional issues. I was able to get the VCF import operation to complete but not without changing the names of the appliances as the VCF Ops Fleet Manager was holding on to the vCenter some where.
Long story short, it was bothering me that when I tried to deploy a new product in VCF that the old vCenter was still showing up. So I went down the rabbit hole of finding it and removing it, was not an easy feat. Thanks to Brock Peterson, I was able to start to look at the APIs on the Fleet manager and begin to figure out where this vCenter record was stored:
VCF Operations 9 Fleet Manager API
Delete Certificates from the VCF Operations 9 Fleet Manager API
Before we begin though, I want to take a moment to say this not the official Broadcom supported removal method and perform this at your own risk. If you are a customer and run into this issue, you should open a ticket with Broadcom support for assistance. Also, even if this was a lab environment for you own testing always take proper backups and snapshots as necessary to ensure if something breaks you have a rollback plan.
First thing you want to do is pull up the VCF Operations Fleet API page, which is located at https://yourfleetmanagerfqdn/api/swagger-ui/index.html. I wasn’t sure where to start with this so I looked through all the different options and it lead me to the Data Center Controller APIs, and I saw there was an option to get all the vCenters listed.
I first had to figure out the dataCenterVmid so that i could run the vCenter Get API,
curl -k -u admin@local:yourpasswordgoeshere -X GET “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/datacenters”
This command showed me that I only had 1 datacenter which was DEFAULT_DC, so now I can find all the vCenters:
curl -k -u admin@local:yourpasswordgoeshere -X GET “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/datacenters/DEFAULT_DC/vcenters”
And this showed that stale vCenter record still showing up, not knowing whether this would cause me any issues in the future. I had to remove it, otherwise it would continue to bug me every time I used my lab. So I’m thinking at this point, easy enough run a delete curl command and we are golden.
I run the command below with my vCenter fqdn and hit Enter.
curl -k -u admin@local:yourpasswordgoeshere -X DELETE “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/datacenters/DEFAULT_DC/vcenters/vcenterserverfqdn”
Well turns out it wasn’t that easy, I get an error that this is still referenced by an environment within VCF and can’t be deleted. Now I’m going back to beginning to figure out what I need to do in order to remove this and look at the Environment Controller API section. There is a command to pull all the environments.
curl -k -u admin@local:yourpasswordgoeshere -X GET “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/environments”
This command shows me roughly 4 different environments, and with me not wanting to sift through all the lines of this GET command, I do a search for each individual environmentId and copy those out to a notepad. Then I decided to run a GET command to pull each environment separately and pipe it to a text file so that I can open them individually and find where this vCenter is.
curl -k -u admin@local:yourpasswordgoeshere -X GET “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/environments/9f3f596a-df23-4003-9c47-915d47ba4f34” >> C:\Environment\9f3f596a-df23-4003-9c47-915d47ba4f34.txt
After running the above command for each different environmentId, I was able to open them up and find that vCenter fqdn and which environment was still showing it as a reference. Inside that environment I also see the old collector that I forgot to remove before re-importing my VCF cluster and this was the record I needed to remove, so I started looking through all the different API sections to figure out which controller handled this but I didn’t find anything at first. That was because I was still on the public-api side, in the top right of the webpage there is a select definition option.
I switched over to the private-internal-api and started comparing what I had in the environmentID text file to determine which parameter would help me remove this record. It took some time looking through all the different sections on this page but I finally located the Product Node Controller section which allowed me to use the nodeVmid and remove this record. By running this command:
curl -k -u admin@local:yourpasswordgoeshere -X DELETE “https://yourfleetmaangerfqdn/lcm/lcops/api/inventory/products/nodes?nodeVmid=d33ea623-effa-4267-bf56-5ce1b0fd1db6”
Now that I was able to remove the reference of that vCenter in the environment, we can re-run the delete vCenter command:
curl -k -u admin@local:yourpasswordgoeshere -X DELETE “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/datacenters/DEFAULT_DC/vcenters/vcenterserverfqdn”
After that is run, I confirmed it was removed by pulling all the vCenters again:
curl -k -u admin@local:yourpasswordgoeshere -X GET “https://yourfleetmaangerfqdn/lcm/lcops/api/v2/datacenters/DEFAULT_DC/vcenters”
That was the fix for this, a simple mistake on my part by not completely cleaning up and old environment led me to multiple hours of API calls but it was a good way to learn the new VCF Operations Fleet Manager APIs.