5. The Art of Possible

The flexible configuration options and processor logic in gecholog provide a toolkit to integrate to your specific use of LLMs. With its open architecture for processors, gecholog allows you to build your own custom processors for your unique requirements. You can write these in any programming language or utilize one of our available processors.

Augment your data

Processors in gecholog can operate synchronously, enhancing both the request and the response in interactions with the LLM service:

  • Add data to your payload (e.g., insert data into your prompts, change settings, etc.).
  • Add or modify headers.
  • Replace parts of your request or response.

Processors can also function in an asynchronous mode to minimize impact on request latency. Asynchronous, in this context, means post-processing after the response has already been sent back to the user.

  • Run external classifiers on requests/responses to include NLP data for improved tracking.
  • Implement counters and measurements.
  • Parse and modify JSON fields, then add them to the log.
  • Create and apply meta-data tags.

Clean your data

In modern architectures, data privacy is a crucial consideration. For gecholog, you can develop processors to:

  • Alter fields before they're forwarded to the target or returned to the requester.
  • Obfuscate PII (Personally Identifiable Information) from logs via post-processing.
  • Selectively filter fields to be included in the final log.

Measure your traffic

gecholog consistently records timestamps and measures the duration of various stages:

  • Time from ingress to outbound.
  • Duration from outbound to inbound.
  • Interval from inbound to egress.
  • Time taken in post-processing.
  • Duration for each processor.

Control traffic

Should you need to establish rules for controlling or blocking traffic before it reaches your target, gecholog request processors are equipped for the task. You have the capability to:

  • Throttle traffic based on token consumption, utilizing the standard library tokencounter processor.
  • Develop a custom content filter tailored to your needs.
  • Design a processor that leverages an external API to respond to specific types of requests.

Use Cases

Here are a couple of examples to describe the Art of Possible. And how you can solve them using gecholog.

PII-cleaner

Remove common patterns like social-security number, phone numbers, addresses etc from all payloads.

Create a processor that parses payloads and replace PII-patterns with * characters. Configure gecholog to run the processor async=true both as a request and response processor.

Content filter

Block requests of certain content before sending it to the LLM-service.

Write a processor that uses 3rd party classification service to classify the content according to your rules (for example identifying non-public company documents). If the classification rating is over a certain threshold, write a json response the control field to block sending traffic to the target. Configure gecholog to run the request processor async=false for the ingress_payload field. See the Custom Processor contentfilter for inspiration.

Use a cache service

Hook up a cache service to provide answers to common questions where the LLM request is not needed.

Your processor would catch the ingress_payload and validate if the cache-service has an answer to propose. If the cache-service has an answer, use the control field send back the response from the cache-service to the user. Configure gecholog to run the processor async=false as a request processor.

Store metrics but no data

You want to track the performance, latency and token cost of your llm-traffic, but you don't want the logs to contain any free-text data.

You don't need a processor to accomplish this. Just configure either the fields_include or the fields_exclude settings in the logger part of the configuration file /app/conf/gl_config.json. These settings determine what fields will be in the final log.

Augment the prompt

You application is reusing similar requests and you want to decouple part of the prompt building from the requestor.

Build a processor to modify the outbound_payload and to expand the prompt according to your templates. Configure gecholog to run your processor as a request processor with modifier=true for the field outbound_payload.

Analyze api calls that belong together

I want to track user sessions, not only each api call.

You don't need a processor to accomplish this, just configure the request from the application to use the built in session_id feature of gecholog. Just send the last session_id in the header for each api call that below together and your logs get the same session_id but different transaction_id.

I want to learn more

Recommended reading to dig deeper into the the world of gecholog