But in the 26ai SQL Language Reference that becomes

and that small change opens a world of opportunity, reducing the number of context switches in certain cases, as shown below. But there are also some ceveats!

The classic set-up:

Suppose you have a table with an identity column as primary key and another table with a foreign key pointing to it. Further, suppose you need PL/SQL to insert records in both that parent and child table. In that case, you’ll need to to use insert … returning in orer to get the PK-value created during that insert in order to use it for the subsequent insert(s) into the child table.

Before 26ai, the retunring_clause was only possible when using the values_clause. If you wanted to use a subquery (an “insert-select”), there was no way to also have a retunring clause.

For example, when using collections in Oracle Apex, it is common to use PL/SQL to select from apex_collections to get the data stored in collections into records in tables.

Before 26ai, this required

1. select …
   from apex_collection
   into :plsql_variables_or_records
   where …

2. insert into destination_parent_table (…)
   values (:plsql_variables_or_records, …)
   returning new ID as :variable_for_parent_key

3. insert into destination_child_table (…)
   values (:variable_for_parent_key, …)

So, that are 3 SQL calls in PL/SQL and therefore also 3 context switches…

In 26ai this can be reduced to

1. insert into destination_parent_table (…) values (
   select … from apex_collection where …
   ) returning new ID as :variable_for_parent_key

2. insert into destination_child_table (…)
   select :variable_for_parent_key, …
   from apex_collections
   where …

So that is

• 1 query less

• 1 context switch less

• less data in PL/SQL => less memory usage

Some code to illustrate this and expose some issues:

drop table if exists src purge;
create table src (id number, val varchar2(30));

-- 1) Multi record insert
   -- Very nice for creating test data or (initial) data loading.
   -- Probably less usefull in application development.
   -- Here it is used to populate a source table (to select from)
insert into src (id, val) values
   (1, 'One'),
   (2, 'Two'),
   (3, 'Three');
commit;

/*
The new way of using subqueries when inserting:
There has always been
   insert into dest (col1, col2)
   select col_a, col_b from src;
It was previously not possible to add a returning clause onto this,
probably since that select potentially returns multiple records.
That fact results in having to do 2 sql-calls, in PL/SQL:
   1 to select the values into variables
   2 to do the insert using the "insert values"-way or working:
         select col_a, col_b
            into :param1, :param2
            from src
            where id = 1;
         insert into dest (col1, col2)
            values (:param1, :param2)
            returning id into :id;
But now, the values(...) can contain a select
*/

-- Testcase for the new functionality:
drop table if exists dest purge;
create table dest (id number GENERATED ALWAYS AS IDENTITY, src_id number, val varchar2(30));

insert into dest (src_id, val) values (
   select id, val from src where id = 1
);
-- Note that it has to be a single row query (or "ORA-01427: single-row subquery returns more than one row" is raised)
-- If you want to insert multiple rows (implying you don't need "returning") just use the old style insert-select.
-- This really only adds value when using "returning into":

set serveroutput on
declare
   dst_id number;
begin
   insert into dest (src_id, val) values (
      select id, val from src where id = 3
   )
   returning new id into dst_id;
   dbms_output.put_line('New dest.id is: '||dst_id);

   insert into dest (src_id, val) values (
      select id, val from src where id = 2
   )
   returning new id into dst_id;
   dbms_output.put_line('New dest.id is: '||dst_id);

   insert into dest (src_id, val) values (
      select id, val from src where id = 1
   )
   returning new id into dst_id;
   dbms_output.put_line('New dest.id is: '||dst_id);

   insert into dest (src_id, val) values (
      select id, val from src where id = 42 --does not exist
   )
   returning new id into dst_id;
   dbms_output.put_line('New dest.id is: '||dst_id);
end;
/

select * from dest;

The caveat is that this returns

So, also when the subquery returns no records, the insert is performed without raising errors…

(Note that in the PL/SQL sample above, the returned PK-value is not used to insert a child record. This part has been intentionally left out since it is not relevent to the new 26ai functionality of the insert statement.)

This is one of the reasons that led me to use Podman instead of Docker Desktop: multiple containers in a pod can talk to each other over localhost:port. The second reason is that https://container-registry.oracle.com/ords/ocr/ba/database also uses Podman in its examples.

Prerequisites

WSL2

You must have WSL2 (Windows Subsystem for Linux) installed. I'm not going to describe that here.

Podman

On my company laptop, the easiest way to install Podman was to run

winget install RedHat.Podman

from a DOS_box.

For those that want a GUI to go along with that, use Podman Desktop:

# Optional:
winget install RedHat.Podman-Desktop

While using that GUI, I found it not showing the things I created with it. Pull an image and it does not appear in the list of images. Create a container and it does not appear as a container. Only by going through a rather obscure menu could I refresh the GUI to show the created items. So I switched to the command line.

Being used to Portainer, I opted to use that instead to check on things. Portainer works with Podman, since Podman exposes the same API as Docker does. The only thing Portainer does not show is Pods, but the Containers in a Pod are all individually available.

Setting up

After having installed Podman (just another Windows CLI program) you have to initialize the default Podman Machine.

This is another reason why I prefer Podman over Docker Desktop: The Podman Machine is a small WSL distribution. Docker Desktop, on the other hand, soiled my default WSL distribution (which happened to be an Oracle Linux) by adding Docker stuff into that.

So only once, you have to run

podman machine init

Running wsl --list shows that a new distribution called podman-machine-default was created.

If you'd like to add that little machine to Windows Terminal, use wsl.exe -d podman-machine-default as command line.

Starting the Podman Machine

While the command line podman program is always available, Podman (the system) needs the Podman Machine to be running (under WSL). So, after the initialization, whenever WSL was shutdown or whenever the Windows host was rebooted, you must start the Podman Machine using

podman machine start

Preparations

Pulling images

Images are like installation packages. While it is possible to create a container without pulling (downloading) the image it will be created from first, I like to pull images separately. Seeing that the one for Oracle 23c FREE is > 9Gb is just one of the reasons.

# 1) The database
podman pull container-registry.oracle.com/database/free:latest
# 2) ORDS (including Apex and Database Actions etc)
podman pull container-registry.oracle.com/database/ords:latest
# 3) Optional: Portianer
podman pull docker.io/portainer/portainer-ce:latest

These downloads end up in the filesystem of the podman-machine-default WSL distribution.

You can list the available images using podman images.

Volumes

A Volume is persisted storage for a Container. It is possible to destroy a Container, but keep its Volume and use that Volume for another Container (even one based on a different Image). It is possible to use ordinary Windows folders as persisted storage, but real Volumes have advantages: performance and security.

The database needs a Volume to store its data files (etc), and ORDS also needs a Volume to store its configuration. (Portainer also uses a Volume, but I didn't bother creating that beforehand.)

podman volume create orcl23c_data
podman volume create ords_config

At this moment those Volumes, orcl23c_data and ords_config, are nothing but names. (And maybe empty directories in the podman-machine-default WSL distribution.)

Use podman volume ls to list the volumes.

The Pod

A Pod is somewhere to put containers together. The main benefit, for me, is that containers in a Pod are like distinct programs on one computer:

• they can communicate with each other over localhost:port.

• they share a single process space

• ...

One drawback of a Podman Pod is that it is the Pod that describes which ports (used by the Container inside the Pod and determined by the images they are based on) are mapped/exposed to which ports on Windows.

It is not possible to add a port later on. (While it is possible to add another container to a pod, later in the pod's life).

From the documentation on https://container-registry.oracle.com/ords/ocr/ba/database/free and https://container-registry.oracle.com/ords/ocr/ba/database/ords I know that only 1521 (DB Listener) and 8181 (http cleartext ORDS) are used. But sometime in the future, I'd like to talk https to ORDS, so I foresee already using 8443 as well.

I will be mapping ports in the pod (1521, 8181 and 8443) to the same ports at the Windows level since I have nothing else using those ports already.

The commend to create the Pod:

podman pod create --name orcl --infra-name pod_orcl_infra -p 1521:1521 -p 8181:8181 -p 8443:8443

Explanation

podman pod create The command to create a pod using Podman. Wow!

--name orcl The pod will be named orcl.

--infra-name pod_orcl_infra Inside every Pod, there is always an extra Container, called the Infrastructure Container, that takes care of pod-relative functionality. This option gives that Container a name, pod_orcl_infra, indicating that it is the Infra Container for Pod orcl. This is mainly handy when looking at Portainer since it will show all Containers individually, not as part of a Pod.

-p 1521:1521, -p 8181:8181, -p 8443:8443 Only those three ports will be exposed. I expose them on the same port numbers. Using for example -p 8521:1521 would expose port 1521 (from inside the container) as port 8521 (in Windows). Using a Windows client, I would then connect using sys/password@localhost:8521/FREE, while inside the Container the listener is listening to port 1521 as usual.

The DB container

The Containers are created using an Image (already pulled) and in our case, they will go in Pod (already created). They will use Volumes that also have already been created.

The ORDS container (which includes Apex) will be created later. First, let's create the DB Container:

podman run -d --pod orcl --name=orcl23c -v orcl23c_data:/opt/oracle/oradata --tz=Europe/Brussels container-registry.oracle.com/database/free:latest

Explanation

podman run The container will not only be created but also immediately run.

-d run in detached mode. Otherwise, you wouldn't get the Windows command prompt back...

--pod orcl Put this Container in a Pod, the one called orcl.

--name=orcl23c Call this Container orcl23c. This name is needed to reference this container later on.

-v orcl23c_data:/opt/oracle/oradata Mount the Volume called orcl23c_data (already created) as directory /opt/oracle/oradata in the Container. That directory will be created based on the Image.

--tz=Europe/Brussels Use that timezone. Otherwise, file timestamps (and DB-time!) in the container would be different from the Windows time.

container-registry.oracle.com/database/free:latest Use this Image to create the Container. So in the container registry of Oracle (located at container-registry.oracle.com) find Image database/free and get its latest version. In our case we have already pulled that image, no further download is needed.

Some things that are not in there:

• I did not set a password (ORACLE_PWD).
  The container provides a way to that later (setPassword.sh). This way, there will be no environment variable exposing the password.
  Using podman secret did not work for me. Presumably, it is a Windows thing regarding quotes etc.

• I did not set ORACLE_CHARACTERSET. The default of AL32UTF8 suits me fine.

• I did not add volumes for additional startup or setup scripts.

Verification

The podman run command takes a surprisingly short amount of time to start the database: just a few seconds.

You can use another DOS prompt to execute podman logs orcl23c. This will show the logs written by the Container called orcl23c. Quite soon this will show

#########################
DATABASE IS READY TO USE!
#########################

This log also reveals that the instance is called FREE and that there is a PDB called FREEPDB1.

Optional:

From inside the Container, you can already connect to the database (remember that no password has been set yet) using the Bequath connection.

To do this, either create a new Windows Terminal profile with as command line podman exec -it orcl23c bash.

Or use that same command straight from the DOS command line.

This tells podman to start an interactive (-i) terminal (-t) (a TTY) to the orcl23c Container and launch the bash command in there. In other words: start the shell in that Container.

In that bash shell, you can now run sqlplus / as sysdba.

Set the passwords

Oracle provides in the container a utility to set the passwords for SYS, SYSTEM and PDBADMIN. In the end, it does nothing more than

sqlplus / as sysdba
      ALTER USER SYS IDENTIFIED BY ...;
      ALTER USER SYSTEM IDENTIFIED BY ...;
      ALTER SESSION SET CONTAINER=FREEPDB1;
      ALTER USER PDBADMIN IDENTIFIED BY ...;

Which you could do in the bash shell shown above. But instead of running bash, you can tell Podman to run that script:

podman exec orcl23c ./setPassword.sh S€cr€tP4ssword

In other words: Dear podman, in Container orcl23c, please execute script ./setPassword.sh which takes as its first parameter my new password: S€cr€tP4ssword. There is no -it in there since we're not interested in interactivity. The output is returned though.

Connect

At this moment it should be possible to use your favorite DB client (on your Windows machine) and connect to the database.

Using the following connection settings in SQL Developer

I can do

Notice the lack of from dual, a new 23c feature.

I guess that supposing I were able to open port 1521 on my laptop to the network, remote machines would also be able to use my DB.

The ORDS Container

Which also includes setting up Apex in the DB!

Preparation

This container requires setting up a conn_string.txt file that contains the connection string to the DB. The directory containing that file will be mounted as a volume. Next, the starting container will read that file, and if all goes well, it will delete the file.

Contrary to the prior volumes created for performance and persistence, this one is just for one-time usage. It is therefore okay to mount an ordinary host folder.

So make a new folder. Its location and name do not really matter but mine was C:\Users\jvdboss1\Documents\Podman\ords_secrets. You'll find that path later on in the Container creation command.

In that folder, I created the expected conn_string.txt file. And that file contained

CONN_STRING=sys/S€cr€tP4ssword@localhost:1521/FREEPDB1

The connection must be for sys since an as sysdba connection is used when installing Apex.

Creating the container

As explained on https://container-registry.oracle.com/ords/ocr/ba/database/ords these are all the preparations needed.

podman run -d --pod orcl --name ords -v C:\Users\jvdboss1\Documents\Podman\ords_secrets:/opt/oracle/variables -v ords_config:/etc/ords/config/ container-registry.oracle.com/database/ords:latest

Explanation

podman run The container will not only be created but also immediately run.

-d run in detached mode. Otherwise, you wouldn't get the Windows command prompt back...

--pod orcl Put this Container in a Pod, the one called orcl. (Which now already has the orcl23c container running inside of it)

--name=ords Call this Container ords. This name is needed to reference this container later on.

-v C:\Users\jvdboss1\Documents\Podman\ords_secrets:/opt/oracle/variables Mount the Windows folder C:\Users\jvdboss1\Documents\Podman\ords_secrets as directory /opt/oracle/variables in the Container. That directory will be created based on the Image.

-v ords_config:/etc/ords/config/ Mount the Volume called ords_config (already created) as directory /etc/ords/config/ in the Container. That directory will be created based on the Image.

container-registry.oracle.com/database/ords:latest Use this Image to create the Container. So in the container registry of Oracle (located at container-registry.oracle.com) find Image database/ords and get its latest version. In our case we have already pulled that image, no further download is needed.

Keeping an eye on things

The installation of Apex takes quite some time, so it is nice to be able to see the output of it.

From a separate dos-prompt, run podman logs -f ords. The -f option makes sure you keep following the logs, not just output the current state of the log, once.

This will tell you

INFO : This container will start a service running ORDS 23.2.3 and APEX 23.1.0.
INFO : CONN_STRING has been found in the container variables file.
INFO : Database connection established.
INFO : Apex is not installed on your database.
INFO : Installing APEX on your DB please be patient.
INFO : You can check the logs by running the command below in a new terminal window:
        docker exec -it orcl tail -f /tmp/install_container.log
WARN : APEX can be installed remotely on PDBs, If you want to install it on a CDB,
       install it directly on the Database and not remotely.

So, it explains there is a command to follow what is being done for the Apex install. But the listed command needs some adapting: We're using Podman instead of Docker and our container is called ords instead of orcl. So,

podman exec -it orcl tail -f /tmp/install_container.log

can be used to tail the Apex installation. Like before, this command will use container ords to execute the tail -f /tmp/install_container.log command.

Eventually (after a couple of minutes) the tail will end, stating

Disconnected from Oracle Database 23c Free Release 23.0.0.0.0 - Develop, Learn, and Run for Free
Version 23.3.0.23.09

In the other DOS-terminal, where the container creation was running, more important information can be seen:

INFO : Use below login credentials to first time login to APEX service:
        Workspace: internal
        User:      ADMIN
        Password:  Welcome_1

And regarding ORDS (which is installed after Apex):

ORDS: Release 23.2 Production on Wed Oct 18 16:04:15 2023

Copyright (c) 2010, 2023, Oracle.

Configuration:
  /etc/ords/config/

The setting named: jdbc.InitialLimit was set to: 10 in configuration: default
INFO : Starting the ORDS services with the following database details:
INFO :   localhost:1521/FREEPDB1.

ORDS: Release 23.2 Production on Wed Oct 18 16:04:17 2023

Copyright (c) 2010, 2023, Oracle.

Configuration:
  /etc/ords/config/

2023-10-18T16:04:17.529Z INFO        HTTP and HTTP/2 cleartext listening on host: 0.0.0.0 port: 8181
2023-10-18T16:04:17.579Z INFO        The document root is serving static resources located in: /etc/ords/config/global/doc_root

And finally, you can surf to http://localhost:8181/ords to get

or to http://localhost:8181/ords/apex to get

Recently the client opted to no longer fetch and treat all those sources themselves but buy the same data from a commercial vendor that does all the cleansing and aligning among the official sources for us (and their other clients).

The result is that tables in the transform schema will have to keep only the data from the new source (it is already in there to be able to check its quality cfr the other sources), while the data from all those other sources needs to be removed.

delete from transform.some_source_tb
where source_type_id not in (20/*new comercial source*/,
                             1/*private data*/,
                             9/*test data*/)

That will do the trick. but since about 50% of the (pretty large!) tables will be removed, it is not very efficient. It is also going to leave a lot of storage space wasted (and the IT crowd is already complaining about how much we use...).

Instead, we came up with a much more complex way of working but with a much better result: The original table is truncated (with a drop of the old storage space) and only the data that needs to be kept gets written back into it.

The scenario in short:

• Create a new table with the same layout as the original one

• Fill the new table with only the data you want to keep

• Truncate the original table

• Add the data from the temporary table back to the original one

• Drop the temporary table

In reality, there are some extra steps in between those to make sure things flow efficiently (read: fast) and data remains correct.

Create a table for temporary use

One would assume that the first two steps can be combined into one single create table as select (CTAS). The reality is that we often run into a shortage of PGA while attempting this ...

But a CTAS that does not select anything does work:

create table temp_tb as select * from some_source_tb WHERE 1=2;

Fill it with the needed data

Instead of deleting data that can be thrown out, this step only copies (i.e. selects) the data that must remain. This is how a delete is turned into the first of a pair of inserts.

The benefit of using CTAS would be that it performs a direct path insert to write the data. To achieve the same performance we use the /*+ append*/ hint.

insert /*+ append*/ into temp_tb
select * from some_source_tb
where source_type_id in (20/*new comercial source*/,
                         1/*private data*/,
                         9/*test data*/);
commit;

Prepare for downtime.

As stated above, we'll truncate the original table. This implies that for some time no data will be available to end-users: downtime. To make this time as short as possible, prepare some things in advance. To make the insertion quick (and correct) we'll

• disable constraints
  You're inserting data that came out of the table, so it will still comply with ordinary constraints. (Only *table-*level constraints might get violated by a delete like this.)

• disable indexes
  It's a big insert, better (quicker) to rebuild indexes after it has been completed.

• disable triggers
  Not only to not spend time running the trigger code but also to keep the data intact: we have classic last-modified-by-and-on-fields and triggers to maintain them. It should be clear that since we are mimicking a delete, the last modification (meta)data should remain untouched.

• Allocate space: an extent large enough for all kept data
  This prevents having to extend the table over and over again while blocks of records are being written.

Disable constraints

Run

select constraint_name, constraint_type, status, validated,
       search_condition_vc, index_name,
       'alter table some_source_tb disable constraint '||constraint_name||';' disable_stmt,
       'alter table some_source_tb enable constraint '||constraint_name||';' enable_stmt
from user_constraints
where table_name='some_source_tb'
order by constraint_type;

to list all constraints on the table, along with generated statements to disable and later re-enable them.

Disable indexes

Run

select index_name, index_type, status,
       domidx_status, domidx_opstatus, funcidx_status,
       'alter index '||index_name||' unusable;' disable_stmt,
       'alter index '||index_name||' rebuild;' enable_stmt
from user_indexes
where table_name='some_source_tb'
order by index_type;

to list all indexes on the table, along with generated statements to disable and later re-enable them.

Disable triggers

Run

select trigger_name, description, status, trigger_body,
       'alter trigger '||trigger_name||' disable;' disable_stmt,
       'alter trigger '||trigger_name||' enable;' enable_stmt,
from user_triggers
where table_name='some_source_tb';

to list all triggers on the table, along with generated statements to disable and later re-enable them.

Allocate Space

Since the temp_tb now contains all data that will eventually end up back in some_source_tb, we can use the size of temp_tb to pre-allocate space for some_source_tb right before the inserting starts.

select segment_name, segment_type, dbms_xplan.format_size(bytes) tab_size
from user_segments
where segment_name='temp_tb'
  and segment_type='TABLE';

will return a tab_size like 123M or 234G. Round it up a bit and note that value down. It will later be used to set the next_extent storage attribute of some_source_tb. But once this operation is finished, you should set it back to its original value, so make sure to also note that orig_next_extent down before changing it:

select table_name, tablespace_name
     , dbms_xplan.format_size(initial_extent) initial_extent
     , dbms_xplan.format_size(next_extent) orig_next_extent
from user_tables where table_name='some_source_tb';

Truncate

After having prepared the statements above, it is time to get started. Once the table is truncated, there is no way back. To reduce downtime, this should be done in one continued succession.

truncate table some_source_tb DROP STORAGE;

The drop storage makes sure that all the space allocated for the big table (of which only a part will remain) is released. The table is now sized only initial_extent seen above. The end-users have lost their data!

Next, set the next_exent. This way, once the initial_extent amount of space is filled, there is one resize operation that will add enough space for the entire dataset.

alter table some_source_tb storage (next <TAB_SIZE>);

Disable triggers, indexes and constraints

Right before inserting, it is time to disable all the functionality that could slow down the inserting (and/or corrupt the data).

alter table some_source_tb disable constraint some_constraint;
alter index some_index unusable;
alter trigger some_trigger disable;

Insert

insert /*+ append parallel 4*/ into some_source_tb
select * from temp_tb;
commit;

The append hint makes sure a direct path load is performed. Depending on your CPU and storage solution the parallel hint might speed up things a bit further. (Adapt the value (4) to your situation or leave it out altogether and Oracle will choose the optimum degree of parallelism for you.)

If you are working on an instance that is not under forcelogging mode, you might want to add the nologging hint. But make sure to ask your DBA about that.

Wrap up

After the commit above, the data is back, but the table can not be used yet: the triggers and constraints need to be re-enabled and to make things perform as usual again, the indexes need to be rebuilt and statistics gathered. Oh, and resetting the next_extend!

alter table some_source_tb storage (next <ORIG_NEXT_EXTENT>);

alter table some_source_tb enable constraint some_constraint;
alter trigger some_trigger enable;
alter index some_index rebuild;

begin
   dbms_stats.gather_table_stats (
      ownname    => user,
      tabname    => 'SOME_SOURCE_TB',
      method_opt => 'for all indexed columns',
      cascade    => TRUE );
end;
/

After this, the downtime is over. End users have their data back (well, only the new source). The only thing left to do is to drop the temp_tb:

drop table temp_db purge;

Hey! What about partition exchange?

Yes, instead of doing an insert select swapping partitions between the two tables would be a much quicker operation and there would be no need to change the extent size etc.

To be able to use alter table tab1 exchange partition p1 with table tab2; the tab1 needs to have at least one partition while tab2 must be not partitioned.

If some_source_tb is partitioned, you could use exchange partition as shown above, but you'll need one temp_tb per partition: the selection of the data to keep becomes a little bit more complex. To create the various temp_tb tables, you can use create table temp_tb1 for exchange with table some_source_tb; .

Alternatively, if some_soure_tb is not partitioned, then you'll need to create temp_tb with a partition.

create table temp_tb
partition by range (source_type_id) (
   partition p_empty values less than (0),
   partition p_keep values less than (100)
)
as
select * from some_table_tb
wherer 1=2;

and the swap becomes

alter table temp_tb exchange partition p_keep with table some_table_tb;

Exchange partition has features to take along indexes as well. You might want to use those features to save downtime: you can build the indexes on temp_tb long before some_table_tb table is truncated.

But I didn't try any of this, because there are no partitions in the DB I generally work with. The reason for that lies in a small but important caveat of Oracle Spatial: The find nearest neighbour function (SDO_NN() to find, for example, the 5 restaurants closest to your current location), will return 5 objects from each partition of the table you are searching in. (see https://docs.oracle.com/en/database/oracle/oracle-database/23/spatl/sdo_nn.html)

I downloaded the Windows version of GraalVM for Java 17: I Surfed to https://www.oracle.com/java/technologies/downloads/. There I chose the tab labeled GraalVM for JDK 17 (so not JDK 20, JDK 17 nor GraalVM for JDK 20)

The subpage indicates that
GraalVM for JDK 17 binaries are free to use in production and free to redistribute, at no cost, under the GraalVM Free Terms and Conditions. (https://www.oracle.com/downloads/licenses/graal-free-license.html)

The reason I chose 17 over 20, is that the 17 is a Long Term Release (until September 2024), while the 20 is not (only until September 2023, next month, when GraalVM for JDK 21 comes out. (It looks like 21 is the next LTS.))
Afterward, I also read somewhere that SQLcl does not support version 20 yet.

I choose Windows to get a download for that platform. That resulted in the file graalvm-jdk-17_windows-x64_bin.zip.

I also found out that I would be needing a plugin to run Javascript in GraalVM. (Since SQLcl can make use of that.)
I hopped over onto https://www.oracle.com/downloads/graalvm-downloads.html. There, I once again indicated version 17 for Windows and somewhere in the middle of the resulting page is a link to download the JavaScript Runtime That resulted in js-installable-jdk-17-windows-amd64-23.0.1.jar. After trying the installation a first time, I found out that the JavaScript Runtime depends on two other plugins: ICU4J Plugin and Tregex Plugin. Downloading these (from that same page as the first plugin) resulted in icu4j-installable-jdk-17-windows-amd64-23.0.1.jar and regex-installable-jdk-17-windows-amd64-23.0.1.jar.

Installing GraalVM

...boils down to unzipping the zip somewhere, adding its bin directory to the PATH and setting the main directory as JAVA_HOME. The zip contains one folder which is best preserved in its entirety (including its name). So, I made folder C:\pf\GraalVM and dragged graalvm-jdk-17.0.8+9.1 from the zip into there, resulting in C:\pf\GraalVM\graalvm-jdk-17.0.8+9.1\bin (and others).

The plugins were still in my downloads folder: C:\Users\jvdboss1\Downloads*-installable-jdk-17-windows-amd64-23.0.1.jar They need to be installed into Graal using its gu command (GraalVM Component Updater), which resides in its bin-folder:

cd C:\pf\GraalVM\graalvm-jdk-17.0.8+9.1\bin
gu -L install "C:\Users\jvdboss1\Downloads\icu4j-installable-jdk-17-windows-amd64-23.0.1.jar"
gu -L install "C:\Users\jvdboss1\Downloads\regex-installable-jdk-17-windows-amd64-23.0.1.jar"
gu -L install "C:\Users\jvdboss1\Downloads\js-installable-jdk-17-windows-amd64-23.0.1.jar"

Running gu list confirms that I have GraalVM, icu4j, regex and js installed, all version 23.0.1.

At this moment, from that bin directory, running java -version returned Java(TM) SE Runtime Environment Oracle GraalVM 17.0.8+9.1 (build 17.0.8+9-LTS-jvmci-23.0-b14) OK :)

Settings

Running echo %path% or just path showed that another Java was there: C:\ProgramData\Oracle\Java\javapath. This is probably something installed by my employer (I don't write any Java programs for my job). It turns out to be a shortcut-folder ("junction") and it contains (only) java.exe, javaw.exe and javaws.exe...
The junction points to its sibling: C:\ProgramData\Oracle\Java\javapath_target_474953
I decided to leave that in the PATH, but put the GraalVM bin-folder first. So now there is ...;C:\pf\GraalVM\graalvm-jdk-17.0.8+9.1\bin;C:\ProgramData\Oracle\Java\javapath;...

While being in the Environment Variables GUI of Windows, I also set JAVA_HOME and gave it the value C:\pf\GraalVM\graalvm-jdk-17.0.8+9.1.

After restarting the command line prompt echo %path% and echo %java_home% confirm that these settings are there. From the default directory, running java -version tells me that GraalVM 17 is being used :)

I already had a SQLcl installation. I went to its bin directory and started it:
cd C:\pf\sqlcl\bin sql /nolog
And it still started. :) Doing show java confirmed that it had picked up the JAVA_HOME setting.

Upgrading SQLcl

Installing a new version of SQLcl is also just about unzipping a zip. From https://www.oracle.com/database/sqldeveloper/technologies/sqlcl/download/ I downloaded sqlcl-23.2.0.178.1027.zip After making sure no more SQLcl was running and all command windows were no longer in the folder, I renamed my existing C:\pf\sqlcl to C:\pf\sqlcl_22_3 (since that was the old version I had).
Next, I dragged the sqlcl folder from the zip to my C:\pf folder.

And that is how you upgrade SQLcl :)

You can always use https://download.oracle.com/otn_software/java/sqldeveloper/sqlcl-latest.zip to directly get the latest version, whatever it is. Ideal for DevOps.

SQLcl falls under the Oracle Free License

I already had C:\pf\sqlcl\bin in my path, so running sql /nolog from anywhere starts SQLcl :)

login.sql

While SQLcl works fine out of the box, obviously, it has many settings which you may want to apply each time. To do that, you can create a glogin.sql or login.sql script, just like for SQL*Plus.

Running show login shows you where that script can reside so that SQLcl picks it up. In my case (having an old fat client installed too) it was:

• C:\Oracle\product\12.2.0\client_1\sqlplus\admin\glogin.sql

• .\login.sql

• C:\Oracle\product\12.2.0\client_1\dbs\login.sql

Notice that the 2nd entry implies that you can have different login.sql scripts per project/environment/..., just by starting SQLcl from a certain directory.

I set up the statusbar, syntax highlighting and DDL generation config. I relaxed the history filter a bit, confirmed the sqlformat (query output), allow empty lines in SQL-scripts and set up the Trivadis Formatter.
Also very important: I explicitly turn autocommit off, but exitcommit on.

That results in the following login.sql script

SET statusbar ON
SET statusbar editmode linecol java cwd encoding username dbid txn timing

SET highlighting ON
--SET highlighting DEFAULT RESET --duh
SET highlighting COMMENT foreground GREEN
--SET highlighting COMMENT background DEFAULT --same as default
SET highlighting STRING foreground MAGENTA
--SET highlighting STRING background DEFAULT --same as default
SET highlighting NUMBER foreground MAGENTA
--SET highlighting NUMBER background DEFAULT --same as default
--SET highlighting PUNCTUATION foreground DEFAULT --same as default
--SET highlighting PUNCTUATION background DEFAULT --same as default
SET highlighting KEYWORD foreground YELLOW
--SET highlighting KEYWORD background DEFAULT --same as default
SET highlighting IDENTIFIER foreground CYAN
--SET highlighting IDENTIFIER background DEFAULT --same as default
--SET highlighting ERROR foreground RED     --same as default
--SET highlighting ERROR background DEFAULT --same as default
--SET highlighting ERROR underline ON       --same as default

SET ddl CONSTRAINTS_AS_ALTER off
SET ddl FORCE off
SET ddl SEGMENT_ATTRIBUTES off
SET ddl SIZE_BYTE_KEYWORD off
SET ddl STORAGE off

-- Include connect and show in the history
-- Note that connect implies a security risk when passords are entered in the connect statement.
-- But normally, only stored connections are used.
-- When exceptionally making a manual connection, don't include the password in the connect statment.
-- Provide it separately. SQLcl will ask for it.
SET history FILTER history clear

SET sqlformat ANSICONSOLE
--SET sqlformet JSON-FORMATTED
--SET sqlformat delimited ; " "

-- Allow empty lines in SQL statements (Even catproc.sql has them)
SET sqlblanklines ON

-- Use command tvdformat to call the Trivadis formatter
script C:\Repos\Trivadis-PLSQL-Formatter-Settings\sqlcl\format.js --register
-- Set up tformat alias to call tvdformat with our own settings to format the buffer
alias tformat=tvdformat * xml=C:\Users\jvdboss1\Documents\Modified_trivadis_formatter.xml arbori=C:\Repos\Trivadis-PLSQL-Formatter-Settings\settings\sql_developer\trivadis_custom_format.arbori ;

SET autocommit OFF
SET exitcommit ON

To have colored text in a field, the magic delimiters @| and |@ need to be used. After the starting delimiter (@|) you need to provide one or more directives to color the text that follows (after a separating space) and close the colored text with |@.

For example

There are only a few colors that can be used: black, green, yellow, blue, magenta, cyan and white.

As stated above, multiple directives can be used. This is necessary to control the display attributes: blinking (fast, slow or off), concealing (on/off), intensity (bold, faint or no change), italic (on/off), negative (on/off).

To get a blinking, italic, red field:

select '@|BLINK_FAST,red,italic This is red data|@' colored_text from dual;

The order of the directives is not important, except for directives of the same kind: From 2 or more colors, only the last one will be in effect. BLINK_FAST followed by BLINK_OFF will result in non-blinking text.

The allowed attributes are

• blink:

  • BLINK_FAST

  • BLINK_SLOW

  • BLINK_OFF

• conceal:

  • CONCEAL_ON (results in hidden letters, i.e. only the background is visible)

  • CONCEAL_OFF

• intensity

  • INTENSITY_BOLD

  • INTENSITY_FAINT

• italic

  • ITALIC

• negative

  • NEGATIVE_OFF

  • NEGATIVE_ON

• reset

  • RESET (undoos all the settings to the left of it. Since the colors are set per field, it should never be necessary)

Instead of using just one (foreground) color, it is of course possible to set a background color too: bg_black, bg_green, bg_yellow, bg_blue, bg_magenta, bg_cyan and bg_white. When setting both foreground and background colors it is best to use fg_black, fg_green, fg_yellow, fg_blue, fg_magenta, fg_cyan and fg_white instead of the version without the fg_ prefix. Since the two notations for the foreground colors are equivalent, the following happens:

Hey Jochen, that yellow looks kinda green! Yup, that is because while SQLcl tells the terminal to show yellow, it is still up to the terminal to decide what that will actually look like. I'm using a "Solarized Dark"-setting in Windows Terminal, hence the strange colors.

There are other things that simply don't work in my terminal: slow and fast blinking texts blink at exactly the same rate. Bold turns the text gray, whatever its original color was...

To find out what combinations give what effect, it is possible to come up with a few queries:

with
   colors as (
      select 'black' value from dual union all
      select 'green'       from dual union all
      select 'yellow'      from dual union all
      select 'blue'        from dual union all
      select 'magenta'     from dual union all
      select 'cyan'        from dual union all
      select 'white'       from dual
   )
 , blink as (
      select 'BLINK_FAST' setting from dual union all
      select 'BLINK_OFF'          from dual union all
      select 'BLINK_SLOW'         from dual union all
      select ''                   from dual --not used
   )
 , conceal as (
      select 'CONCEAL_OFF' setting from dual union all
      select 'CONCEAL_ON'          from dual union all
      select ''                    from dual --not used
   )
 , intensity as (
      select 'INTENSITY_BOLD' setting from dual union all
      select 'INTENSITY_FAINT'        from dual union all
      select ''                       from dual --not used
   )
 , italic as (
      select 'ITALIC' setting from dual union all
      select ''               from dual --not used
   )
 , negative as (
      select 'NEGATIVE_OFF' setting from dual union all
      select 'NEGATIVE_ON'          from dual union all
      select ''                     from dual --not used
   )
/* , reset as (
    select 'RESET' setting from dual union all
    select ''              from dual --not used
RESET removes all attributes (italic, blinking,...) to the left of it, leaving only those that follow
)*/
 , underline as (
      select 'UNDERLINE' setting from dual union all
      select 'UNDERLINE_DOUBLE' from dual union all
      select 'UNDERLINE_OFF' from dual union all
      select '' from dual /*no attribute*/
   )
select lpad(c.value, 7, ' ')
       || nvl2(bl.setting, ',' || bl.setting, bl.setting)
       || nvl2(co.setting, ',' || co.setting, co.setting)
       || nvl2(iy.setting, ',' || iy.setting, iy.setting)
       || nvl2(it.setting, ',' || it.setting, it.setting)
       || nvl2(ne.setting, ',' || ne.setting, ne.setting)
       || nvl2(un.setting, ',' || un.setting, un.setting) as color_setting
     , '@|' || c.value
       || nvl2(bl.setting, ',' || bl.setting, bl.setting)
       || nvl2(co.setting, ',' || co.setting, co.setting)
       || nvl2(iy.setting, ',' || iy.setting, iy.setting)
       || nvl2(it.setting, ',' || it.setting, it.setting)
       || nvl2(ne.setting, ',' || ne.setting, ne.setting)
       || nvl2(un.setting, ',' || un.setting, un.setting)
       || ' Test Me|@' as show_color
  from      colors    c
 cross join blink     bl
 cross join conceal   co
 cross join intensity iy
 cross join italic    it
 cross join negative  ne
 cross join underline un
 order by c.value desc nulls first
        , bl.setting nulls first, co.setting nulls first
        , iy.setting nulls first, it.setting nulls first
        , ne.setting nulls first, un.setting nulls first;

The query above gives all (foreground) colors, combined with all attributes. This turns out to result in 6048 combinations.

It can be extended to also use foreground and background colors:

with
   foreground as (
      select 'fg_black' value from dual union all
      select 'fg_green'       from dual union all
      select 'fg_yellow'      from dual union all
      select 'fg_blue'        from dual union all
      select 'fg_magenta'     from dual union all
      select 'fg_cyan'        from dual union all
      select 'fg_white'       from dual
   )
 , background as (
      select 'bg_black' value from dual union all
      select 'bg_green'       from dual union all
      select 'bg_yellow'      from dual union all
      select 'bg_blue'        from dual union all
      select 'bg_magenta'     from dual union all
      select 'bg_cyan'        from dual union all
      select 'bg_white'       from dual
   )
 , blink as (
      select 'BLINK_FAST' setting from dual union all
      select 'BLINK_OFF'          from dual union all
      select 'BLINK_SLOW'         from dual union all
      select ''                   from dual --not used
   )
 , conceal as (
      select 'CONCEAL_OFF' setting from dual union all
      select 'CONCEAL_ON'          from dual union all
      select ''                    from dual --not used
   )
 , intensity as (
      select 'INTENSITY_BOLD' setting from dual union all
      select 'INTENSITY_FAINT'        from dual union all
      select ''                       from dual --not used
   )
 , italic as (
      select 'ITALIC' setting from dual union all
      select ''               from dual --not used
   )
 , negative as (
      select 'NEGATIVE_OFF' setting from dual union all
      select 'NEGATIVE_ON'          from dual union all
      select ''                     from dual --not used
   )
/* , reset as (
    select 'RESET' setting from dual union all
    select ''              from dual --not used
RESET removes all attributes (italic, blinking,...) to the left of it, leaving only those that follow
)*/
 , underline as (
      select 'UNDERLINE' setting from dual union all
      select 'UNDERLINE_DOUBLE' from dual union all
      select 'UNDERLINE_OFF' from dual union all
      select '' from dual /*no attribute*/
   )
select lpad(b.value || ',' || f.value, 21, ' ')
       || nvl2(bl.setting, ',' || bl.setting, bl.setting)
       || nvl2(co.setting, ',' || co.setting, co.setting)
       || nvl2(iy.setting, ',' || iy.setting, iy.setting)
       || nvl2(it.setting, ',' || it.setting, it.setting)
       || nvl2(ne.setting, ',' || ne.setting, ne.setting)
       || nvl2(un.setting, ',' || un.setting, un.setting) as color_setting
     , '@|' || b.value || ',' || f.value
       || nvl2(bl.setting, ',' || bl.setting, bl.setting)
       || nvl2(co.setting, ',' || co.setting, co.setting)
       || nvl2(iy.setting, ',' || iy.setting, iy.setting)
       || nvl2(it.setting, ',' || it.setting, it.setting)
       || nvl2(ne.setting, ',' || ne.setting, ne.setting)
       || nvl2(un.setting, ',' || un.setting, un.setting)
       || ' Test Me|@' as show_color
  from      foreground f
 cross join background b
 cross join blink      bl
 cross join conceal    co
 cross join intensity  iy
 cross join italic     it
 cross join negative   ne
 cross join underline  un
 --where f.vlaue != b.value --same background and foreground color sounds useless, but with attributes set it just might display something
 order by f.value asc nulls first, b.value desc nulls first --desc+asc so the equal bg and fg is not first.
        , bl.setting nulls first, co.setting nulls first
        , iy.setting nulls first, it.setting nulls first
        , ne.setting nulls first, un.setting nulls first;

This gives 42336 combinations, but many of them are useless or look exactly the same as other combinations.

It is possible to use multiple colors in one field:

Coloring can be used to dynamically highlight values:

Remember that the ANSI color codes (that is what the @| is inserting to the string) can only be added to string values.

Note: Generic highlighting (for all queries, without writing @| in them) is better achieved using
set sqlformat ansiconsole -config=highlight.json
but that is a completely separate topic.

Links

The Trivadis Formatter and instructions on how to install it can be found at
https://github.com/Trivadis/plsql-formatter-settings

That Jeff Smith's blog post regarding SQLcl:
https://www.thatjeffsmith.com/archive/tag/sqlcl/
There is no better source than the source itself!

But now it is throwing connection refused errors. (PS this blog is a subdomain of that website, but blog.jochenvandenbossche.dev is just a CNAME DNS-record pointing to hashnode.network, so that still works.)

I kind of guessed the Let's Encrypt certificate wasn't renewed for some reason. I can still connect with SSH etc. When I tried to force a refresh from the command line, there was some kind of message regarding an outdated version of OpenSSL (the python package). When I tried to update that, I noticed all of this was still using Python 2.7... That is when I decided to start a new webserver from scratch.

So, I stopped an instance that I wasn't really using and spun up a new one. Apparently, the Always free instances are Ampères these days? The 6GB of RAM won't hurt. I created some users (one in my name, one called webserver).

I installed NginX, according to Install the NginX webserver in Oracle Linux (docs.oracle.com) but diverted a bit to also apply the security settings described in NginX and PHP-FPM what my permission should be (www.getpagespeed.com).

I added a separate /etc/nginx/conf.d/jochenvandenbossche.dev.conf, setting up a separate server to handle requests for dev.jochenvandenbossche.dev, www.jochenvandenbossche.dev and jochenvandenbossche.dev. My DNS was already pointing www.jochenvandenbossche.dev and jochenvandenbossche.dev to the old server (the one that started refusing connections recently); I added a new A-record pointing dev.jochenvandenbossche.dev to the public IP address of the newly created Ampère.

From the server itself, curl 127.0.0.1 and curl 141.*.*.* (the public IP) both resulted in the default NginX page, as expected: the separate server only serves domain name-based requests. And, lo and behold, curl dev.jochenvandenbossche.dev resulted in the separate little index.html that I set up. great!

On my laptop, I have WSL running an Ubuntu. Doing curl dev.jochenvandenbossche.dev from there also resulted in the expected html. Nice, everything works! Joy, joy joy!

So eventually I opened a browser (Vivaldi in my case) to surf to http://dev.vandenbossche.dev and I got a connection refused error. Bummer.

At first, I thought it was a DNS problem. Doing an nslookup from both Ubuntu (under WSL) and a command prompt indeed showed my ISPs DNS being used from Ubuntu and Google.dns from Windows. But both returned the same correct IP address.

After calling out on Reddit (see this topic) it turned out to (probably) be SHTS, a browser (+webservder) security feature. The Network tab of the Developers Console in Chrome showed my HTTP request resulting in a 307 Internal Rederect to an HTTPS request...

And since nothing regarding HTTPS is set up on that NginX yet, it will be blocked.

Opening the chrome://net-internals/#hsts "page" and querying jochenvandenbossche.dev (the top-level domain) shows that SHTS is being applied, including for subdomains:

So, while the old server itself (jochenvandenbossche.dev) can't serve anything anymore, it is still capable of requiring the usage of HTTPS on all its subdomains, even on dev.jochenvandenbossche.dev, which didn't exist when it was set up.

So, now I hope that when I set up Let's Encrypt it will take into account the other two domains as well. I guess/hope that changing the DNS-records for the existing domains (pointing them to the new server) should do the trick.

ORDER BY is so easy to add at the end of a query. And I expect you also know about ASC end DESC. Good, but that is not all there is to say about sorting (in Oracle).

Do you know that there is a session parameter that influences the result of an ORDER BY enormously? Since it is a session parameter, you as a developer can not know what value the user will have set it to (knowingly, but usually quite unknowingly). Therefore you, as a developer, should set this parameter whenever you write code that depends on the sort order of a query. Unless you want two distinct runs of the same code to yield different results. Or unless you want the code that behaves perfectly in your environment to crash on the environment of your colleague / the tester / the code reviewer / production / the end user / ...

The parameter in question is NLS_SORT. Note that, according to the Oracle documentation, NLS_SORT "is derived from NLS_LANGUAGE". What this means is that whenever you change NLS_LANGUAGE, you will also have changed NLS_SORT. Therefore make sure to set NLS_LANGUAGE first and then set NLS_SORT:

alter session set NLS_LANGUAGE='ENGLISH';
alter session set NLS_SORT='BINARY';

So what is it all about?

Sorting per letter.

Let's sort a table with 4 records, each of 1 character: dot, underscore, lowercase a, uppercase A.

Let's try with 12 settings of NLS_SORT. Since some variants turn out to give the same result, they only get one column in the output:

• UNICODE_BINARY gives the same result as BINARY.
  I guess that might be different on databases that are not in a Unicode character set and where the text expression being sorted contains Unicode values.

• The X-versions of the languages (DUTCH vs XDUTCH / FRENCH and XFRENCH / GERMAN and XGERMAN / WEST_EUROPEAN vs XWESTEUROPEAN) give the same result

Did you know that in German lowercase comes before uppercase?

Do you think this is all you need to know about sorting? Well, look what happens when sorting longer strings... (below)

Did you know that trying to set NLS_SORT to 'AMERICAN' or 'ENGLISH' gives an error? I guess that it is because of the A in ASCII.

Sorting strings with punctuation (e.g. "codes")

Here I sort eight 3-letter codes composed of A, a, B, b and the underscore.

Note how in DUTCH, XDUTCH, FRENCH, XFRENCH, LATIN, WEST_EUROPEAN and XWEST_EUROPEAN there is an _ below an a, while the test above shows _ being sorted above the a! And at the same time there is also an _ above an A, meaning that there are A and a between two _ ?!?!!??!?

What is going on? Well, My first thought was: The _ are removed from the string before sorting. Let's visualise that by removing the _ from the sorted list (DUTCH):

Clearly, even that is not fully correct: Why is AAA below aa when A is above a in our first test?
My second guess is that not only are the _ removed, the (first) ordering is done without looking at case (so all letters uppercased / or all letters lowercased).
And then, if there are strings with equal sort order, they are compared again using/including case.
It is normal/acceptable that a short string (like AA) comes before a longer string that starts with that short string (like AAA) So both aa and AA end up before aaa and AAA: aa and AA with order 1; aaa and AAA as order 2.
In the second step AA gets 1.1 and aa 1.2, while AAA gets 2.1 and aaa 2.2.

GERMAN and XGERMAN also show letters between underscores: the same thing is happening as with (X)DUTCH, (X)FRENCH, LATIN and (X)WEST_EUROPEAN.
The only difference with those is that in German lowercase comes before uppercase (as seen before).

Let's add in the ., resulting in twelve strings (and let's remove half of the seemingly equal sorting schemes)

What you should learn from this:

1. Sorting is not as straightforward as it seems!

2. If you have trouble sorting (for example) names because of uppercase/lowercase, try one of the language-specific sorts: They tend to ignore case until sorting equal strings, giving you exactly what you probably want.

3. If you need to sort any kinds of codes (which may include punctuation like _ and .), stay away from the language-specific sorts: they give unexpected results. BINARY, EEC_EURO and PUNCTUATION are all quite different but "logical" nevertheless.