Question: How to rename database name within Oracle XE, a free DBMS with 4GB limited storage? --" (The default instance database is XE, so I gotta change it to VMS).

Answer: I assumed that you've known how to launch sqlplus within command prompt and how to log in with SYSDBA privileges.
Let's see...

1. Backup the database.

2. Mount the database after a clean shutdown:
SHUTDOWN IMMEDIATE
STARTUP MOUNT

3. Invoke the DBNEWID utility (nid) specifying the new DBNAME from the command line using a user with SYSDBA privilege:
nid TARGET=sys/password@XE DBNAME=VMS

Assuming the validation is successful the utility prompts for confirmation before performing the actions. Typical output may look something like:

C:\oracle\920\bin>nid TARGET=sys/password@XE DBNAME=VMS
DBNEWID: Release 9.2.0.3.0 - Production
Copyright (c) 1995, 2002, Oracle Corporation. All rights reserved.

Connected to database XE (DBID=1024166118)

Control Files in database:
C:\ORACLE\ORADATA\XE\CONTROL01.CTL
C:\ORACLE\ORADATA\XE\CONTROL02.CTL
C:\ORACLE\ORADATA\XE\CONTROL03.CTL

Change database ID and database name XE to VMS? (Y/[N]) => Y

Proceeding with operation
Changing database ID from 1024166118 to 1317278975
Changing database name from XE to VMS
Control File C:\ORACLE\ORADATA\XE\CONTROL01.CTL - modified
Control File C:\ORACLE\ORADATA\XE\CONTROL02.CTL - modified
Control File C:\ORACLE\ORADATA\XE\CONTROL03.CTL - modified
Datafile C:\ORACLE\ORADATA\XE\SYSTEM01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\UNDOTBS01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\CWMLITE01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\DRSYS01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\EXAMPLE01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\INDX01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\ODM01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\TOOLS01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\USERS01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\XDB01.DBF - dbid changed, wrote new name
Datafile C:\ORACLE\ORADATA\XE\TEMP01.DBF - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\XE\CONTROL01.CTL - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\XE\CONTROL02.CTL - dbid changed, wrote new name
Control File C:\ORACLE\ORADATA\XE\CONTROL03.CTL - dbid changed, wrote new name

Database name changed to VMS.
Modify parameter file and generate a new password file before restarting.
Database ID for database VMS changed to 1317278975.
All previous backups and archived redo logs for this database are unusable.
Shut down database and open with RESETLOGS option.
Succesfully changed database name and ID.
DBNEWID - Completed succesfully.

4. This ORACLE_SID in windows registry should be modified to approriate name, in this case is VMS

5. Shutdown the database:
SHUTDOWN IMMEDIATE

6. Modify the DB_NAME parameter in the initialization parameter file. The startup will result in an error but proceed anyway.
STARTUP MOUNT
ALTER SYSTEM SET DB_NAME=VMS SCOPE=SPFILE;
SHUTDOWN IMMEDIATE

7. Create a new password file:
orapwd file=C:\oraclexe\app\oracle\product\10.2.0\server\database\pwdVMS.ora password=password entries=10

8. Rename the SPFILE to match the new DBNAME, eg. SPFILEVMS.ORA
Mostly the default location is at here C:\oraclexe\app\oracle\product\10.2.0\server\dbs\

9. (i) If you are using Windows you must recreate the service so the correct name and parameter file are used:
oradim -delete -sid XE
oradim -new -sid VMS -intpwd password -startmode a -pfile C:\oraclexe\app\oracle\product\10.2.0\server\dbs\SPFILEVMS.ora
(ii) If you are using UNIX/Linux simply reset the ORACLE_SID environment variable:
ORACLE_SID=VMS; export ORACLE_SID

10. Alter the listener.ora and tnsnames.ora setting to match the new database name and restart the listener:
lsnrctl reload

11. Modify this filename 'initXE.ora' become 'initVMS.ora' (default location is C:\oraclexe\app\oracle\product\10.2.0\server\database\)

12. Modify the content as below:
SPFILE='C:\oraclexe\app\oracle\product\10.2.0\server\dbs/spfileVMS.ora'

13. Open the database with RESETLOGS:
STARTUP MOUNT
ALTER DATABASE OPEN RESETLOGS;

14. Verify the new database with DBID and NAME as below:
SELECT DBID, NAME
FROM V$DATABASE;

15. Backup the database. Because you reset the online redo logs, the old backups and archived logs are no longer usable in the current incarnation of the database.

Okay, that's for renaming database id and name. And remember to edit these 2 files 'StartDB.bat' and 'StopDb.bat' at C:\oraclexe\app\oracle\product\10.2.0\server\BIN\
Find this word 'XE' (by default) through those 2 files and replace them with 'VMS', this will ensure everything's okay when we start and stop the database (windows service) later

Yes, only that.. so cumbersome..

Reference: Oracle and Oracle-Base