Calling OCI Generative AI from Python: Models, Serving Modes, and Response Handling

My first OCI Generative AI Python experiments reinforced something I see in many cloud SDKs: the model call itself is simple; the real work is understanding the request hierarchy, serving mode, model identifiers, message representation, and nested response types.

One detail matters immediately: the inference API lives in oci.generative_ai_inference. It is not a separate oci.generative_ai_chat package.

1. Load OCI Configuration

import oci

config = oci.config.from_file(
    file_location="~/.oci/config",
    profile_name="DEFAULT",
)

oci.config.validate_config(config)

The profile typically contains tenancy, user, fingerprint, region, and private-key location. I keep compartment and model identifiers outside source code:

import os

COMPARTMENT_ID = os.environ["OCI_COMPARTMENT_ID"]
MODEL_ID = os.environ["OCI_GENAI_MODEL_ID"]

2. Create the Inference Client

from oci.generative_ai_inference import (
    GenerativeAiInferenceClient
)

client = GenerativeAiInferenceClient(
    config=config
)

This client is for runtime inference operations such as chat and embeddings. I keep OCI management APIs separate from application inference code.

3. Choose a Serving Mode

from oci.generative_ai_inference.models import (
    OnDemandServingMode
)

serving_mode = OnDemandServingMode(
    model_id=MODEL_ID
)

The explicit serving-mode object makes it clear whether the application is using an on-demand model or a dedicated endpoint.

4. Build a Generic Chat Message

from oci.generative_ai_inference.models import (
    UserMessage,
    TextContent,
)

message = UserMessage(
    content=[
        TextContent(
            text=(
                "Explain why an Oracle execution plan "
                "might change after statistics are gathered."
            )
        )
    ]
)

5. Create GenericChatRequest

from oci.generative_ai_inference.models import (
    GenericChatRequest
)

chat_request = GenericChatRequest(
    messages=[message],
    temperature=0.2,
    max_tokens=1000,
)

For database and infrastructure questions I prefer a lower temperature because consistency matters more than creative wording.

6. Wrap the Request in ChatDetails

from oci.generative_ai_inference.models import ChatDetails

details = ChatDetails(
    compartment_id=COMPARTMENT_ID,
    serving_mode=serving_mode,
    chat_request=chat_request,
)

The object hierarchy is worth understanding:

ChatDetails
 ├── compartment_id
 ├── serving_mode
 └── chat_request
      └── messages
           └── UserMessage
                └── TextContent

7. Call the Model and Parse the Nested Response

response = client.chat(details)

chat_response = response.data.chat_response
choice = chat_response.choices[0]
assistant_message = choice.message

text = assistant_message.content[0].text

print(text)

The result is not a flat string. Once function calling is added, this distinction becomes even more important because the assistant message may contain tool_calls in addition to normal content.

8. Hide OCI-Specific Nesting Behind an Adapter

class OciChatProvider:
    def __init__(self, client, compartment_id, model_id):
        self.client = client
        self.compartment_id = compartment_id
        self.model_id = model_id

    def chat(self, text):
        request = GenericChatRequest(
            messages=[
                UserMessage(
                    content=[TextContent(text=text)]
                )
            ],
            temperature=0.2,
            max_tokens=1200,
        )

        details = ChatDetails(
            compartment_id=self.compartment_id,
            serving_mode=OnDemandServingMode(
                model_id=self.model_id
            ),
            chat_request=request,
        )

        response = self.client.chat(details)

        msg = (
            response.data.chat_response
            .choices[0]
            .message
        )

        return msg.content[0].text

The rest of the program can now use a much simpler interface:

provider = OciChatProvider(
    client,
    COMPARTMENT_ID,
    MODEL_ID,
)

answer = provider.chat(
    "What is the difference between "
    "STARTUP MOUNT and ALTER DATABASE OPEN?"
)

9. Capture Useful OCI Errors

from oci.exceptions import ServiceError

try:
    answer = provider.chat("Explain DBMS_XPLAN.")
except ServiceError as exc:
    print("status:", exc.status)
    print("code:", exc.code)
    print("message:", exc.message)
    print("request-id:", exc.request_id)
    raise

Those fields help separate IAM/authentication issues from region, compartment, model availability, and malformed-request problems.

10. Keep Provider Objects at the Edge

Application
    ↓ plain request
OCI adapter
    ↓ SDK request objects
GenerativeAiInferenceClient
    ↓
OCI Generative AI
    ↓ SDK response objects
OCI adapter
    ↓ plain application response