This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

Integration guides and API references for every Cobalt engine.

Each engine is documented the same way: what it does, how to generate a client, how to connect to a server, then a page per streaming call and a generated API reference.

If you are starting from scratch, follow Transcribe’s getting started guide — the other engines use the same connection and streaming patterns.

1 - Speech Recognition & Transcription

Demos

1.1 - Transcribe

Low latency, high accuracy on-prem / on-cloud solutions to your ASR needs.

Web Demo

Cobalt’s Transcribe engine is a state-of-the-art speech recognition system. Cobalt Transcribe supports two different DNN architectures:

  • Hybrid models combine separately tunable Acoustic Models, Lexicons, and Language Models, making them highly customizable for specific use cases. Hybrid models support extremely low-latency partial results.

  • End-to-end models go straight from sounds to words in the same DNN. They tend to be more accurate for general use cases, particularly for systems in which sub-second response time is not required.

Cobalt Transcribe is a highly flexible system that can run on-premise, in your private cloud, or fully embedded on your device. Your data – both the audio and the transcripts – never leave your control.

The SDK is based on a gRPC API and client code can be easily generated for different languages using the proto definition, including C++, C#, Go, Java and Python, and can add support for more languages as required.

Once running, Transcribe’s API provides a method to which you can stream audio. This audio can either be from a microphone or a file. We recommend uncompressed WAV as the encoding, but support other formats such as MP3, ulaw etc.

Cubic setup

Transcribe’s API provides a number of options for returning the speech recognition results. The results are passed back using Google’s protobuf library, allowing them to be handled natively by your application. Transcribe can estimate its confidence in the transcription result at the word or utterance level, along with timestamps of the words. Confidence scores are in the range 0-1. Transcribe’s output options are described below.

Automatic Transcription Results

The simplest result that Transcribe returns is its best guess at the transcription of your audio. Transcribe recognizes the audio you are streaming, listens for the end of each utterance, and returns the speech recognition result.

Transcribe maintains its transcriptions in an N-best list, i.e. is the top N transcriptions from the recognizer. The best ASR result is the first entry in this list.

Click here to see an example json representation of Transcribe’s N-best list with utterance-level confidence scores
{
  "alternatives": [
    {
      "transcript": "TOMORROW IS A NEW DAY",
      "confidence": 0.514
    },
    {
      "transcript": "TOMORROW IS NEW DAY",
      "confidence": 0.201
    },
    {
      "transcript": "TOMORROW IS A <UNK> DAY",
      "confidence": 0.105
    },
    {
      "transcript": "TOMORROW IS ISN'T NEW DAY",
      "confidence": 0.093
    },
    {
      "transcript": "TOMORROW IS A YOUR DAY",
      "confidence": 0.087
    }
  ],
}

A single stream may consist of multiple utterances separated by silence. Transcribe handles each utterance separately.

For longer utterances, it is often useful to see the partial speech recognition results while the audio is being streamed. For example, this allows you to see what the ASR system is predicting in real-time while someone is speaking. Transcribe supports both partial and final ASR results.

Confusion Network

A Confusion Network is a form of speech recognition output that’s been turned into a compact graph representation of many possible transcriptions, as here:

Confusion Network Example

Note that <eps> in this representation is silence.

Click here to see an example json representation of this Confusion Network object, with time stamps and word-level confidence scores
{
  "cnet": {
    "links": [
      {
        "duration": "1.350s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "0s"
      },
      {
        "duration": "0.690s",
        "arcs": [
          {
            "word": "TOMORROW",
            "confidence": 1.0
          }
        ],
        "startTime": "1.350s"
      },
      {
        "duration": "0.080s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.040s"
      },
      {
        "duration": "0.168s",
        "arcs": [
          {
            "word": "IS",
            "confidence": 0.892
          },
          {
            "word": "<eps>",
            "confidence": 0.108
          }
        ],
        "startTime": "2.120s"
      },
      {
        "duration": "0.010s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.288s"
      },
      {
        "duration": "0.093s",
        "arcs": [
          {
            "word": "A",
            "confidence": 0.620
          },
          {
            "word": "<eps>",
            "confidence": 0.233
          },
          {
            "word": "ISN'T",
            "confidence": 0.108
          },
          {
            "word": "THE",
            "confidence": 0.039
          }
        ],
        "startTime": "2.298s"
      },
      {
        "duration": "0.005s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.391s"
      },
      {
        "duration": "0.273s",
        "arcs": [
          {
            "word": "NEW",
            "confidence": 0.661
          },
          {
            "word": "<UNK>",
            "confidence": 0.129
          },
          {
            "word": "YOUR",
            "confidence": 0.107
          },
          {
            "word": "YOU",
            "confidence": 0.102
          }
        ],
        "startTime": "2.396s"
      },
      {
        "duration": "0s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "2.670s"
      },
      {
        "duration": "0.420s",
        "arcs": [
          {
            "word": "DAY",
            "confidence": 0.954
          },
          {
            "word": "TODAY",
            "confidence": 0.044
          },
          {
            "word": "<UNK>",
            "confidence": 0.002
          }
        ],
        "startTime": "2.670s"
      },
      {
        "duration": "0.270s",
        "arcs": [
          {
            "word": "<eps>",
            "confidence": 1.0
          }
        ],
        "startTime": "3.090s"
      }
    ]
  }
}

Formatted output

Many speech recognition systems typically output raw words exactly as spoken, without any formatting which can improve intelligibility. Cobalt Transcribe’s customizable formatting suite enables a variety of intelligent formatting options:

  • Capitalizing the first letter of the utterance
  • Numbers: “cobalt’s atomic number is twenty seven” -> “Cobalt’s atomic number is 27”
  • TrueCasing: “the iphone was launched in two thousand and seven” -> “The iPhone was launched in 2007”
  • Ordinals: “summer solstice is twenty first june” -> “Summer solstice is 21st June”

1.1.1 - Getting Started

How to get a Transcribe Server running on your system

Using Cobalt Transcribe

  • A typical Transcribe release, provided as a compressed archive, will contain a linux binary (transcribe-server) for the required native CPU architecture, appropriate Dockerfile and models.

  • Cobalt Transcribe runs either locally on linux or using Docker.

  • Cobalt Transcribe will serve the Transcribe GRPC API on port 2727. A web demo will be enabled on port 8080.

  • To quickly try out Transcribe, first start the server as shown below and open the web demo at http://localhost:8080 in your browser to send live microphone input or upload an audio file for transcription. You can also use the SDK to use Transcribe from within your application or just command line.

Running Transcribe Server Locally on Linux

./transcribe-server
  • By default, the binary assumes the presence of a configuration file, located in the same directory, named: transcribe-server.cfg.toml. A different config file may be specified using the --config argument.

Running Transcribe Server as a Docker Container

To build and run the Docker image for Transcribe, run:

docker build -t cobalt-transcribe .
docker run -p 2727:2727 -p 8080:8080 cobalt-transcribe

How to Get a Copy of the Transcribe Server and Models

Please contact us for finding a product release or Transcribe model best suited to your requirements.

The demo release you will receive is a compressed archive (tar.bz2) and is structured accordingly:

release.tar.bz2
├── COPYING
├── README.md
├── transcribe-server
├── transcribe-server.cfg.toml
├── cobalt.license.key
├── Dockerfile
├── models
│   └── en_US-16khz
├── formatters
│   └── en_US-16khz
│
└── cobalt.license.key [ provided separately, needs to be copied over ]
  • The README.md file contains information about this release and instructions for how to start the server on your system.

  • The transcribe-server is the server program which is configured using the transcribe-server.cfg.toml file.

  • The Dockerfile can be used to create a container that will let you run Transcribe server on non-linux systems such as MacOS and Windows.

  • The models and formatters directories contain your speech recognition and text formatting models. The content of these directories will depend on the models you downloaded.

System Requirements

Cobalt Transcribe runs on Linux. You can run it directly as a linux application, or using Docker.

You can evaluate the product on Windows or Linux using Docker Desktop but we would not recommend this setup for use in a production environment.

A Cobalt Transcribe release typically includes a single Transcribe model together with binaries and config files. The general purpose Transcribe models take up to 4GB disk space, and need a minimum of 4GB RAM when evaluating locally. For production workloads, we recommend configuring containerized applications with each instance allocated with 4 CPUs and 8GB RAM.

Cobalt Transcribe runs on x86_64 CPUs. We also support Arm64 CPUs, including processors such as the Graviton (AWS c7g EC2 instances). Transcribe is significantly more cost effective to run on C7g instances compared to similarly sized Intel or AMD processors, and we can provide you an Arm64 release on request.

To integrate Cobalt Transcribe into your application, please follow the next steps to Generate the SDK in a language of your choice.

1.1.2 - Running Quick Tests

How to use our prebuilt client application to test Cobalt Transcribe.

In the release package you received, we have bundled a prebuilt application, a command-line client, that you can use to connect to the server you now have running. You can use this client to send files to the server for saving the transcription, either as a stream of text, or as JSON with additional information.

Obtaining the Client Application

In the release package of Cobalt Transcribe, you should find a folder called transcribe-client, that contains client binaries for multiple platforms.

transcribe-client/
└── bin
    ├── darwin_amd64
    │   └── transcribe-client
    └── linux_amd64
        └── transcribe-client

This client is implemented in Go, and the code is available for reference.

If you need to build the client yourself, instead of using the pre-packaged version in the release, you can install it using:

go install github.com/cobaltspeech/examples-go/transcribe/transcribe-client@latest

Running The Client

After you follow the Getting Started instructions, you should have Cobalt Transcribe server running and listening on port 2727 on the local machine.

You can run the transcribe client in various ways:

# Transcribe a file using the default address of the Cobalt Transcribe Server (localhost:2727)
transcribe-client recognize input.wav
# Transcribe a file, pointing to some other server address
transcribe-client recognize input.wav --server host:port
# Transcribe a file using the default server address, and save output as a JSON file.
transcribe-client recognize input.wav --output-json output.json
# (Advanced use): List information about the models available on the server.
transcribe-client list
# (Advanced use): Transcribe a file, get word level timestamps and confidences.
transcribe-client recognize input.wav --output-json output.json --recognition-config '{"enable_word_details": true}'

For more details on the recognition-config struct, please see the API spec

# Getting usage information
transcribe-client --help

1.1.3 - Generating SDKs

Gives instructions about how to generate an SDK for your project from the proto API definition.
  • APIs for all Cobalt’s services are defined as a protocol buffer specification or simply a proto file and be found in the cobaltspeech/proto github repository.

  • The proto file allows a developer to auto-generate client SDKs for a number of different programming languages. Step by step instructions for generating your own SDK can be found below.

  • We provide pre-generated SDKs for a couple of languages. You can choose to use these instead of generating your own. These are listed here along with instructions on how to install / import them into your projects.

Pre-generated SDKs

Golang

import transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
  • An example client using the above repo can be found here.

Python

  • Pre-generated SDK files for Python can be found in the cobaltspeech/py-genproto repo

  • The Python SDK depends on Python >= 3.5. You may use pip to perform a system-wide install, or use virtualenv for a local install. To use it in your Python project, install it:

pip install --upgrade pip
pip install "git+https://github.com/cobaltspeech/py-genproto"

Generating SDKs

Step 1. Installing buf

  • To work with proto files, we recommend using buf, a user-friendly command line tool that can be configured generate documentation, schemas and SDK code for different languages.
# Latest version as of March 14th, 2023.

COBALT="${HOME}/cobalt"
  mkdir -p "${COBALT}/bin"

VERSION="1.15.1"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
  curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and adding to $PATH.

chmod +x "${COBALT}/bin/buf"
  export PATH="${PATH}:${COBALT}/bin"
brew install bufbuild/buf/buf

Step 2. Getting proto files

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Change this to where you want to clone the repo to.
PROTO_REPO="${COBALT}/git/proto"

git clone https://github.com/cobaltspeech/proto "${PROTO_REPO}"

Step 3. Generating code

  • The cobaltspeech/proto repo provides a buf.gen.yaml config file to get you started with a couple of languages.

  • Other plugins can be added to the buf.gen.yaml file to generate SDK code for more languages.

  • To generate the SDKs, simply run the following (assuming the buf binary is in your $PATH)

cd "${PROTO_REPO}"

# Removing any previously generated files.
rm -rf ./gen

# Generating code for all proto files inside the `proto` directory.
buf generate proto
  • You should now have a folder called gen inside ${PROTO_REPO} that contains the generated code. The latest version of the transcribe API is v5. You can import / include / copy the generated files into your projects as per the conventions of different languages.
gen
├── ... other languages ...
└── py
└── cobaltspeech
├── ... other services ...
└── transcribe
└── v5
├── transcribe_pb2_grpc.py
├── transcribe_pb2.py
└── transcribe_pb2.pyi
gen
├── ... other languages ...
└── go
   ├── cobaltspeech
   │ ├── ...
   │   └── transcribe
   │   └── v5
   │   ├── transcribe_grpc.pb.go
   │   └── transcribe.pb.go
   └── gw
   └── cobaltspeech
   ├── ...
   └── transcribe
   └── v5
   └── transcribe.pb.gw.go
gen
├── ... other languages ...
└── cpp
   └── cobaltspeech
   ├── ...
   └── transcribe
   └── v5
   ├── transcribe.grpc.pb.cc
   ├── transcribe.grpc.pb.h
   ├── transcribe.pb.cc
   ├── transcribe.pb.h
   ├── transcribe.pb.validate.cc
   └── transcribe.pb.validate.h

Step 4. Installing gPRC and protobuf

  • A couple of gRPC and protobuf dependencies are required along with the code generated above. The method of installing them depends on the programming language being used.
  • These dependencies and the most common way of installing/ / including them are listed below for some chosen languages.
# It is encouraged to this inside a python virtual environment

# to avoid creating version conflicts for other scripts that may

# be using these libraries.

pip install --upgrade protobuf
pip install --upgrade grpcio
pip install --upgrade google-api-python-client
go get google.golang.org/protobuf
go get google.golang.org/grpc
go get google.golang.org/genproto
# More details on grpc installation can be found at:

# https://grpc.io/docs/languages/cpp/quickstart/

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Latest version as of 14th March, 2023.

VERSION="v1.52.0"
GRPC_REPO="${COBALT}/git/grpc-${VERSION}"

git clone \
 --recurse-submodules --depth 1 --shallow-submodules \
 -b "${VERSION}" \
 https://github.com/grpc/grpc ${GRPC_REPO}

cd "${GRPC_REPO}"
mkdir -p cmake/build

# Change this to where you want to install libprotobuf and libgrpc.

# It is encouraged to install gRPC locally as there is no easy way to

# uninstall gRPC after you’ve installed it globally.

INSTALL_DIR="${COBALT}"

cd cmake/build
cmake \
 -DgRPC_INSTALL=ON \
 -DgRPC_BUILD_TESTS=OFF \
 -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
 ../..

make -j
make install

1.1.4 - Connecting to the Server

Describes how to connect to a running Cobalt Transcribe server instance.
  • Once you have your Transcribe server up and running, and have generated the SDK for your project, you can connect to a running instance of Transcribe server, by “dialing” a gRPC connection.

  • First, you need to know the address where the server is running: e.g. host:grpc_port. By default, this is localhost:2727 and should be logged to the terminal when you first start Transcribe server as grpcAddr:

2023/03/15 07:54:01 info  {"license":"Copyright © 2015--present. Cobalt Speech and Language, Inc.  For additional details, including information about open source components used in this software, please see the COPYING file bundled with this program."}
2023/03/15 07:54:01 info  {"msg":"reading config file","path":"transcribe-server.cfg.toml"}
2023/03/15 07:54:01 info  {"msg":"version","server":"v5.3.5-b70948b","built":"2023-03-14"}
2023/03/15 07:54:01 info  {"msg":"server initializing"}
2023/03/15 07:54:01 info  {"msg":"server started","grpcAddr":"[::]:8027","httpApiAddr":"[::]:8030","httpOpsAddr":"[::]:8031"}
  • The default binding address and port for the gRPC / http server (bundled webpage demo) can be configured in the transcribe-server config file.

Default Connection

  • The following code snippet connects to the server and queries its version. It connects to the server using an “insecure” gRPC channel. This would be the case if you have just started up a local instance of Transcribe server without TLS enabled.
import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress  = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribepb.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribepb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Connect with TLS

  • In our recommended setup for deployment, TLS is enabled in the gRPC connection, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

  • The following snippets show how to connect to a Transcribe Server that has TLS enabled. They use the cobalt’s self-hosted demo server at demo.cobaltspeech.com:2727, but you obviously use your own server instance.

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "demo.cobaltspeech.com:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	transcribepb "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress  = "demo.cobaltspeech.com:2727"
		connectTimeout = 10 * time.Second
	)

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout)
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(creds),
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribepb.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribepb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

  • In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your Transcribe server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

  • Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS.

  • The following snippets show how to present client certificates when setting up the credentials. These could then be used in the same way as the examples above to connect to a TLS enabled server.

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string 
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)
package main

import (
	// ...

	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	// ..
)

func main() {
	// ...

	// Root PEM certificate for validating self-signed server certificate
	var rootCert []byte

	// Client PEM certificate and private key.
	var certPem, keyPem []byte

	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
		fmt.Printf("unable to use given caCert\n")
		os.Exit(1)
	}

	clientCert, err := tls.X509KeyPair(certPem, keyPem)
	if err != nil {
		fmt.Printf("unable to use given client certificate and key: %v\n", err)
		os.Exit(1)
	}

	tlsCfg := tls.Config{
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{clientCert},
	}

	creds := credentials.NewTLS(&tlsCfg)

	// ...
}

1.1.5 - Streaming Recognition

Describes how to stream audio to Transcribe server.
  • The following example shows how to transcribe a audio stream using Transcribe’s StreamingRecognize request. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

  • We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here.

  • The examples below use a WAV file as input to the streaming recognition. We will query the server for available models and use the first model to transcribe the speech.

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Set the recognition config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = transcribe.RecognitionConfig(
    model_id=modelID,
)

# Open audio file.
audio = open("test.wav", "rb")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
for resp in client.StreamingRecognize(stream(cfg, audio)):
    processResponse(resp)
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"sync"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribe "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribe.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribe.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &transcribe.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting the first model.
	cfg := &transcribe.RecognitionConfig{
		ModelId: modelResp.Models[0].Id,
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting recognition.
	err = StreamingRecognize(ctx, client, cfg, audio, printTranscript)
	if err != nil {
		fmt.Printf("failed to run streaming recognition: %v\n", err)
		os.Exit(1)
	}
}

// StreamingRecognize wraps the bidirectional streaming API for performing
// speech recognition. It sets up recognition using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to cubic
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// As results are received from Transcribe server, they will be sent to the
// provided handlerFunc.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
//
// This function returns only after all results have been passed to the
// resultHandler.
func StreamingRecognize(
	ctx context.Context,
	client transcribe.TranscribeServiceClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	handlerFunc func(*transcribe.StreamingRecognizeResponse),
) error {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingRecognize(ctx)
	if err != nil {
		return err
	}

	// There are two concurrent processes going on. We will create a new
	// goroutine to read audio and stream it to the server.  This goroutine
	// will receive results from the stream.  Errors could occur in both
	// go routines.  We therefore setup a channel, errCh, to hold these
	// errors. Both go routines are designed to send up to one error, and
	// return immediately. Therefore we use a buffered channel with a
	// capacity of two.
	errCh := make(chan error, 2)

	// start streaming audio in a separate goroutine
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
			// if sendAudio encountered io.EOF, it's only a
			// notification that the stream has closed.  The actual
			// status will be obtained in a subsequent Recv call, in
			// the other goroutine below.  We therefore only forward
			// non-EOF errors.
			errCh <- err
		}

		wg.Done()
	}()

	// Receive results from the stream.
	for {
		in, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}

		if err != nil {
			errCh <- err
			break
		}

		handlerFunc(in)
	}

	wg.Wait()

	select {
	case err := <-errCh:
		// There may be more than one error in the channel, but it is
		// very likely they are related (e.g. connection reset causing
		// both the send and recv to fail) and we therefore return the
		// first error and discard the other.
		return err
	default:
		return nil
	}
}

// printTranscript is a callback function given to StreamingRecognize method to
// print results that are returned though the gRPC stream.
func printTranscript(resp *transcribe.StreamingRecognizeResponse) {
	if resp.Error != nil {
		fmt.Printf("\n[ERROR] server returned an error: %v\n", resp.Error)
		return
	}

	hyp := resp.Result.Alternatives[0]
	startTime := float32(hyp.StartTimeMs) / 1000.0
	endTime := startTime + float32(hyp.DurationMs)/1000.0

	if resp.Result.IsPartial {
		fmt.Printf("\r[%0.2f:%0.2f] %s", startTime, endTime, hyp.TranscriptFormatted)
	} else {
		fmt.Printf("[%0.2f:%0.2f] %s\n\n", startTime, endTime, hyp.TranscriptFormatted)
	}
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream transcribe.TranscribeService_StreamingRecognizeClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the recognition config
	if err := stream.Send(&transcribe.StreamingRecognizeRequest{
		Request: &transcribe.StreamingRecognizeRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&transcribe.StreamingRecognizeRequest{
				Request: &transcribe.StreamingRecognizeRequest_Audio{
					Audio: &transcribe.RecognitionAudio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}
			if err != io.EOF {
				return err
			}
			return nil

		}
	}
}

Streaming from microphone

  • Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

  • The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe
import subprocess

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = transcribe.RecognitionConfig(
    model_id=modelID,
	audio_format_raw=transcribe.AudioFormatRAW(
      encoding="AUDIO_ENCODING_SIGNED",
      bit_depth=16,
      byte_order="BYTE_ORDER_LITTLE_ENDIAN",
      sample_rate=m.attributes.sample_rate,
      channels=1,
    )
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above.
cmd = f"rec --no-show-progress -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 -"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
  	print(f"[ERROR] failed to read audio from mic stream: {err}")

print("\n[INFO] recording from microphone ... Press ctrl + c to exit\n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
try:
  for resp in client.StreamingRecognize(stream(cfg, audio)):
      processResponse(resp)
except KeyboardInterrupt:
	# Stop streaming when ctrl + c pressed.
	pass
except Exception as err:
	print(f"[ERROR] failed to stream audio: {err}")

audio.close()
mic.kill()
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"os/signal"
	"strings"
	"sync"
	"syscall"

	"golang.org/x/sync/errgroup"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribe "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribe.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribe.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &transcribe.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &transcribe.RecognitionConfig{
		ModelId: m.Id,
		AudioFormat: &transcribe.RecognitionConfig_AudioFormatRaw{
			AudioFormatRaw: &transcribe.AudioFormatRAW{
				Encoding:   transcribe.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  transcribe.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above.
	args := fmt.Sprintf("--no-show-progress -t raw -r %d -e signed -b 16 -L -c 1 -", m.Attributes.SampleRate)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording from microphone ... Press ctrl + c to exit\n")

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	eg.Go(func() error { return StreamingRecognize(ctx, client, cfg, audio, printTranscript) })

	// Also using a routine to monitor for interrupts.
	eg.Go(func() error {
		const maxInterrupts = 10
		interrupt := make(chan os.Signal, maxInterrupts)
		signal.Notify(interrupt, os.Interrupt, syscall.SIGTERM)

		<-interrupt
		cancel()

		return ctx.Err()
	})

	if err := eg.Wait(); err != nil && !errors.Is(err, ctx.Err()) {
		fmt.Printf("failed to run streaming recognition: %v\n", err)
	}
}

// StreamingRecognize wraps the bidirectional streaming API for performing
// speech recognition. It sets up recognition using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to cubic
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// As results are received from Transcribe server, they will be sent to the
// provided handlerFunc.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
//
// This function returns only after all results have been passed to the
// resultHandler.
func StreamingRecognize(
	ctx context.Context,
	client transcribe.TranscribeServiceClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	handlerFunc func(*transcribe.StreamingRecognizeResponse),
) error {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingRecognize(ctx)
	if err != nil {
		return err
	}

	// There are two concurrent processes going on. We will create a new
	// goroutine to read audio and stream it to the server.  This goroutine
	// will receive results from the stream.  Errors could occur in both
	// go routines.  We therefore setup a channel, errCh, to hold these
	// errors. Both go routines are designed to send up to one error, and
	// return immediately. Therefore we use a buffered channel with a
	// capacity of two.
	errCh := make(chan error, 2)

	// start streaming audio in a separate goroutine
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
			// if sendAudio encountered io.EOF, it's only a
			// notification that the stream has closed.  The actual
			// status will be obtained in a subsequent Recv call, in
			// the other goroutine below.  We therefore only forward
			// non-EOF errors.
			errCh <- err
		}

		wg.Done()
	}()

	// Receive results from the stream.
	for {
		in, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}

		if err != nil {
			errCh <- err
			break
		}

		handlerFunc(in)
	}

	wg.Wait()

	select {
	case err := <-errCh:
		// There may be more than one error in the channel, but it is
		// very likely they are related (e.g. connection reset causing
		// both the send and recv to fail) and we therefore return the
		// first error and discard the other.
		return err
	default:
		return nil
	}
}

// printTranscript is a callback function given to StreamingRecognize method to
// print results that are returned though the gRPC stream.
func printTranscript(resp *transcribe.StreamingRecognizeResponse) {
	if resp.Error != nil {
		fmt.Printf("\n[ERROR] server returned an error: %v\n", resp.Error)
		return
	}

	hyp := resp.Result.Alternatives[0]
	startTime := float32(hyp.StartTimeMs) / 1000.0
	endTime := startTime + float32(hyp.DurationMs)/1000.0

	if resp.Result.IsPartial {
		fmt.Printf("\r[%0.2f:%0.2f] %s", startTime, endTime, hyp.TranscriptFormatted)
	} else {
		fmt.Printf("[%0.2f:%0.2f] %s\n\n", startTime, endTime, hyp.TranscriptFormatted)
	}
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream transcribe.TranscribeService_StreamingRecognizeClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the recognition config
	if err := stream.Send(&transcribe.StreamingRecognizeRequest{
		Request: &transcribe.StreamingRecognizeRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&transcribe.StreamingRecognizeRequest{
				Request: &transcribe.StreamingRecognizeRequest_Audio{
					Audio: &transcribe.RecognitionAudio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}
			if err != io.EOF {
				return err
			}
			return nil

		}
	}
}

1.1.6 - Recognition Configurations

Describes how to configure requests to Transcribe server.
  • An in-depth explanation of the methods, data structures and types in the auto-generated SDKs can be found in the API Reference section. The sub-section on the RecognitionConfig object is particularly important here. This page discusses the common combinations of values set in RecognitionConfig sent to the server.

  • First, here’s a quick overview of the fields in RecognitionConfig.

FieldRequiredDefaultDescription
model_idYes-Unique ID of the model to use.
audio_format_rawYes for raw audio-Can be used to specify the details of raw audio samples recorded from a microphone stream, for example.
audio_format_headeredNoUNSPECIFIEDCan be used when audio has a self-describing header such as WAV, FLAC, MP3, OPUS etc. If not set, transcribe-server will try to auto-detect the audio encoding from the header.
selected_audio_channelsNo[0] (mono)Specifies which channels of a multi-channel audio file to be transcribed, each as their own individual audio stream.
selected_audio_channelsNo[0] (mono)Specifies which channels of a multi-channel audio file to be transcribed, each as their own individual audio stream.
audio_time_offset_msNo0Can be used to indicate that the audio being streamed to the recognizer is offset from the original stream by the provided duration in milliseconds. This offset will be added to all timestamps in results returned by the recognizer.
enable_confusion_networkNofalseToggles the inclusion of a confusion network, consisting of multiple alternative transcriptions. The specified model must also support confusion networks for this field to be populated.
metadataNo""Can be used to send any custom metadata associated with the audio being sent.The server may record this metadata when processing the request. The server does not use this field for any other purpose.
contextNonilCan be used to provide any context information that can aid speech recognition, such as probable phrases or words that may appear in the recognition output or even out of vocabulary words for the model being used. Currently all context information must first be pre-compiled via the CompileContext().

Use cases

Transcribing Headered Files

  • The most basic use case is getting a formatted transcript for a headered audio file such foo.wav. This would simply need a config such as the following:
{
    "model_id": "1",
}
  • Transcribe will return one or more results depending on partial result frequency, end points in speech etc. each of which would like the following:
{
  "error": null,
  "result": {
    "alternatives": [
      {
        "transcript_formatted": "Tomorrow is a new day.",
        "transcript_raw": "TOMORROW IS A NEW DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.870,
      },
      {
        "transcript_formatted": "Tomorrow is a you day.",
        "transcript_raw": "TOMORROW IS A YOU DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.130,
      }
      // ...
      // Other alternative hypotheses.
      // ...
    ]
  }
}
  • If some sort of non-fatal error was encountered, then Transcribe will populate the error field. One such case maybe sending audio sampled at a lower sample rate than what the model is configured for (e.g. sending 8 kHz audio to a 16 kHz model):
{
  "error": {
    "message": "potential accuracy loss: input sample rate (8000) is lower than required (16000)"
  },
  "results": {
    // ...
    // Results
    // ...
  },
}

Transcribing Raw Audio Stream

  • For transcribing raw audio streams, such as those coming in from a live microphone, the details of the audio samples such as their sampling rate, encoding etc. must be specified in the RecognitionConfig like so:
{
    "model_id": "1",
    "audio_format_raw": {
      encoding="SIGNED",
      bit_depth=16,
      byte_order="LITTLE_ENDIAN",
      sample_rate=16000,
      channels=1,
    }
}
  • For various other encoding formats for raw samples, check AudioFormatRaw in the API specification.

Getting Word-level Details

  • If you need to know the word-level details such as word timestamps, to align subtitles with a video, for example, then you can use the following config to enable those word-level timestamps.
{
    "model_id": "1",
    "enable_word_details": true
}
  • Each alternative hypothesis in the returned results will have a word_details field containing details for both formatted and raw words:
{
  "error": null,
  "result": {
    "alternatives": [
      {
        "transcript_formatted": "Tomorrow is a new day.",
        "transcript_raw": "TOMORROW IS A NEW DAY",
        "start_time_ms": 180,
        "duration_ms": 1425,
        "confidence": 0.870,
        "word_details": {
          "formatted": [
            { "word": "Tomorrow", "confidence": 1.0, "start_time_ms": 180, "duration_ms": 800 },
            { "word": "is", "confidence": 1.0, "start_time_ms": 980, "duration_ms": 120 },
            { "word": "a", "confidence": 1.0, "start_time_ms": 1100, "duration_ms": 120 },
            { "word": "new", "confidence": 0.870, "start_time_ms": 1220, "duration_ms": 210 },
            { "word": "day.", "confidence": 1.0, "start_time_ms": 1450, "duration_ms": 155 },          
          ],
          "raw": [
            { "word": "TOMORROW", "confidence": 1.0, "start_time_ms": 180, "duration_ms": 800 },
            { "word": "IS", "confidence": 1.0, "start_time_ms": 980, "duration_ms": 120 },
            { "word": "A", "confidence": 1.0, "start_time_ms": 1100, "duration_ms": 120 },
            { "word": "NEW", "confidence": 0.870, "start_time_ms": 1220, "duration_ms": 210 },
            { "word": "DAY", "confidence": 1.0, "start_time_ms": 1450, "duration_ms": 155 },          
          ],
        }
      },
      // ...
      // Other alternative hypotheses.
      // ...
    ]
  }
}

Getting Confusion Networks

  • For applications that need more than the one-best transcription, the most comprehensive and detailed results are found in the confusion network. Please refer to the in-depth confusion network documentation to see what is included.

  • To enable the confusion network, the following config can be used:

{
    "model_id": "1",
    "enable_confusion_network": true
}
  • The confusion network will be accessible at the cnet field in the results returned:
{
  "error": null,
  "result": {
    "alternatives": [
        {
          "transcript_formatted": "Tomorrow is a new day.",
          "transcript_raw": "TOMORROW IS A NEW DAY",
          "start_time_ms": 180,
          "duration_ms": 1425,
          "confidence": 0.870,
        },
        {
          "transcript_formatted": "Tomorrow is a you day.",
          "transcript_raw": "TOMORROW IS A YOU DAY",
          "start_time_ms": 180,
          "duration_ms": 1425,
          "confidence": 0.130,
        }
        // ...
        // Other alternative hypotheses.
        // ...
    ],
    "cnet": {
      "links": [
        { 
          "start_time_ms": 180,
          "duration_ms": 800,
          "arcs": [
            { "word": "TOMORROW", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 980,
          "duration_ms": 120,
          "arcs": [
            { "word": "IS", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 1100,
          "duration_ms": 120,
          "arcs": [
            { "word": "A", "confidence": 1.0 }
          ]
        },
        { 
          "start_time_ms": 1220,
          "duration_ms": 210,
          "arcs": [
            { "word": "NEW", "confidence": 0.870 },
            { "word": "YOU", "confidence": 0.130 }
          ]
        },
        { 
          "start_time_ms": 1450,
          "duration_ms": 155,
          "arcs": [
            { "word": "DAY", "confidence": 1.0 }
          ]
        }        
      ]
    }
  }
}

1.1.7 - Recognition Context

Describes how to provide context information for aiding speech recognition.
  • Cobalt Transcribe allows users to send context information with a recognition request which may aid the speech recognition. For example, if you have a list of names that you want to make sure the Transcribe model recognizes correctly, with the correct spelling, then you may provide the list in the form of a RecognitionContext object along with the RecognitionConfig before streaming data.

  • Transcribe models allow different sets of “context tokens” each of which can be paired with a list of words or phrases. For example, a Transcribe model may have a context token for airport names, and you can provide a list of airport names you want to be recognized correctly for this context token. Likewise, models may also be configured with tokens for “contact list names”, “menu items”, “medical jargon” etc.

To ensure that there is no added latency in processing the list of words or phrases during a recognition request, we have a API method called CompileContext() that allows the user to compile the list into a compact, efficient format for passing to the StreamingRecognize() method.

Compiling Recognition Context

  • The following snippet shows an example of how to compile context data and then send it during a recognition request.
import grpc
import cobaltspeech.transcribe.v5.transcribe_pb2_grpc as stub
import cobaltspeech.transcribe.v5.transcribe_pb2 as transcribe

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.TranscribeServiceStub(channel)

# Get server version.
versionResp = client.Version(transcribe.VersionRequest())
print(versionResp)

# Get list of models on the server. 
modelResp = client.ListModels(transcribe.ListModelsRequest())
for model in modelResp.models:
    print(model)


# Select a model ID from the list above. Going with the first model
# in this example. Also printing list of allowed context tokens.
m = modelResp.models[0]
print(f"context tokens =  {m.attributes.context_info.allowed_context_tokens}")

# Let's say this model has an allowed context token called "airport_names" and
# we have a list of airport names that we want to make sure the recognizer gets
# right. We compile the list of names using the CompileContext(), save the compiled
# data and send it back with subsequent recognize requests to customize and improve
# the results.
#
# More typically, general models have a "catch-all" token called "unk:default" which
# can be used to boost the probabilities of any type word, as well as add words that
# are not in the model's vocabulary.
phrases = ["NARITA", "KUALA LUMPUR INTERNATIONAL", "ISTANBUL ATATURK", "LAGUARDIA"]
token = m.attributes.context_info.allowed_context_tokens[0]  # "unk:default" 

compileReq = transcribe.CompileContextRequest(
    model_id=m.id,
    token=token,
    phrases=[ transcribe.ContextPhrase(text=t) for t in phrases ],
)

# Sending compilation request. 
compiledResp = client.CompileContext(compileReq)

# Saving the compiled result for later use; note this compiled data is only
# compatible with the model whose ID was provided in the CompileContext call
compiledContexts = []
compiledContexts.append(compiledResp.context)

# Set the recognition config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = transcribe.RecognitionConfig(
    model_id=m.id,
    context=transcribe.RecognitionContext(compiled=compiledContexts),
)

# Open audio file.
audio = open("test.wav", "rb")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield transcribe.StreamingRecognizeRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield transcribe.StreamingRecognizeRequest(
          audio=transcribe.RecognitionAudio(data=data),
        )
        data = audio.read(bufferSize)

# We also define a callback function to execute for each response.
# The example below just prints the formatted transcript to stdout.
def processResponse(resp):
    result = resp.result
    hyp = result.alternatives[0]                    # 1-best hypothesis.
    transcript = hyp.transcript_formatted           # Formatted transcript.
    start = hyp.start_time_ms / 1000.0              # Converting to seconds.
    end = start + hyp.duration_ms / 1000.0          # Converting to seconds.
    newLine = "\r" if result.is_partial else "\n\n" # Will not move to new line for partial results.
    print(f"[{start:0.2f}:{end:0.2f}] {transcript}", end=newLine)

# Streaming requests to the server.
for resp in client.StreamingRecognize(stream(cfg, audio)):
    processResponse(resp)
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"log"
	"os"
	"sync"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	transcribe "github.com/cobaltspeech/go-genproto/cobaltspeech/transcribe/v5"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := transcribe.NewTranscribeServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &transcribe.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &transcribe.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Select a model ID from the list above. Going with the first model
	// in this example. Also printing list of allowed context tokens.
	m := modelResp.Models[0]
	fmt.Printf("context tokens =  %v\n", m.Attributes.ContextInfo.AllowedContextTokens)

	// Let's say this model has an allowed context token called "airport_names" and
	// we have a list of airport names that we want to make sure the recognizer gets
	// right. We compile the list of names using the CompileContext(), save the compiled
	// data and send it back with subsequent recognize requests to customize and improve
	// the results.
	//
	// More typically, general models have a "catch-all" token called "unk:default" which
	// can be used to boost the probabilities of any type word, as well as add words that
	// are not in the model's vocabulary.
	phrases := []string{"NARITA", "KUALA LUMPUR INTERNATIONAL", "ISTANBUL ATATURK", "LAGUARDIA"}
	token := m.Attributes.ContextInfo.AllowedContextTokens[0] // "unk:default"

	compileReq := &transcribe.CompileContextRequest{
		ModelId: m.Id,
		Token:   token,
		Phrases: make([]*transcribe.ContextPhrase, 0, len(phrases)),
	}

	for _, t := range phrases {
		compileReq.Phrases = append(compileReq.Phrases, &transcribe.ContextPhrase{
			Text: t,
		})
	}

	// Sending compilation request.
	compiledResp, err := client.CompileContext(context.Background(), compileReq)
	if err != nil {
		log.Fatal(err)
	}

	// Saving the compiled result for later use; note this compiled data is only
	// compatible with the model whose ID was provided in the CompileContext call
	compiledContexts := []*transcribe.CompiledContext{compiledResp.Context}

	// Set the recognition config. We don't set the audio format and let the
	// server auto-detect the format from the file header.
	cfg := &transcribe.RecognitionConfig{
		ModelId: m.Id,
		Context: &transcribe.RecognitionContext{
			Compiled: compiledContexts,
		},
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting recognition.
	err = StreamingRecognize(ctx, client, cfg, audio, printTranscript)
	if err != nil {
		fmt.Printf("failed to run streaming recognition: %v\n", err)
		os.Exit(1)
	}
}

// StreamingRecognize wraps the bidirectional streaming API for performing
// speech recognition. It sets up recognition using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to cubic
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// As results are received from Transcribe server, they will be sent to the
// provided handlerFunc.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
//
// This function returns only after all results have been passed to the
// resultHandler.
func StreamingRecognize(
	ctx context.Context,
	client transcribe.TranscribeServiceClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	handlerFunc func(*transcribe.StreamingRecognizeResponse),
) error {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingRecognize(ctx)
	if err != nil {
		return err
	}

	// There are two concurrent processes going on. We will create a new
	// goroutine to read audio and stream it to the server.  This goroutine
	// will receive results from the stream.  Errors could occur in both
	// go routines.  We therefore setup a channel, errCh, to hold these
	// errors. Both go routines are designed to send up to one error, and
	// return immediately. Therefore we use a buffered channel with a
	// capacity of two.
	errCh := make(chan error, 2)

	// start streaming audio in a separate goroutine
	var wg sync.WaitGroup
	wg.Add(1)
	go func() {
		if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
			// if sendAudio encountered io.EOF, it's only a
			// notification that the stream has closed.  The actual
			// status will be obtained in a subsequent Recv call, in
			// the other goroutine below.  We therefore only forward
			// non-EOF errors.
			errCh <- err
		}

		wg.Done()
	}()

	// Receive results from the stream.
	for {
		in, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}

		if err != nil {
			errCh <- err
			break
		}

		handlerFunc(in)
	}

	wg.Wait()

	select {
	case err := <-errCh:
		// There may be more than one error in the channel, but it is
		// very likely they are related (e.g. connection reset causing
		// both the send and recv to fail) and we therefore return the
		// first error and discard the other.
		return err
	default:
		return nil
	}
}

// printTranscript is a callback function given to StreamingRecognize method to
// print results that are returned though the gRPC stream.
func printTranscript(resp *transcribe.StreamingRecognizeResponse) {
	if resp.Error != nil {
		fmt.Printf("\n[ERROR] server returned an error: %v\n", resp.Error)
		return
	}

	hyp := resp.Result.Alternatives[0]
	startTime := float32(hyp.StartTimeMs) / 1000.0
	endTime := startTime + float32(hyp.DurationMs)/1000.0

	if resp.Result.IsPartial {
		fmt.Printf("\r[%0.2f:%0.2f] %s", startTime, endTime, hyp.TranscriptFormatted)
	} else {
		fmt.Printf("[%0.2f:%0.2f] %s\n\n", startTime, endTime, hyp.TranscriptFormatted)
	}
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream transcribe.TranscribeService_StreamingRecognizeClient,
	cfg *transcribe.RecognitionConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the recognition config
	if err := stream.Send(&transcribe.StreamingRecognizeRequest{
		Request: &transcribe.StreamingRecognizeRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&transcribe.StreamingRecognizeRequest{
				Request: &transcribe.StreamingRecognizeRequest_Audio{
					Audio: &transcribe.RecognitionAudio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}
			if err != io.EOF {
				return err
			}
			return nil

		}
	}
}

1.1.8 - Hybrid vs End-to-End Models

Differences between gen-1 and gen-2 models, and choosing the right model type

Cobalt’s Transcribe engine supports two types of models:

  • Hybrid (gen-1) - A Hybrid model consists of a sequence of independent models that when chained together can convert audio to words. This type of model no longer produces state-of-the art accuracy but still remains dominant in many commercial ASR applications.
  • End-to-End (gen-2) - An End-to-End (E2E) model is mostly a single large neural network that can convert audio directly to text transcripts (or something very close that requires little additional processing)

A hybrid model can be viewed as a sequence of several different models glued together in a particular sequence to convert audio to text. The cascade of models will (1) convert audio to features based on the amount of energy in different frequency ranges, (2) use a neural network to predict context-dependent sounds (phones) that are present for every ~10 milliseconds of audio, (3) convert the context-dependent phones to context-endependent phones (ex: the ’th’ sound in ’the’ would be an example phone), (4) convert the sequence of phones into a sequence of words using a lexicon model that is a manually curated dictionary of words and the expected sounds/pronunciations for each word, (5) convert the identified possible sequences of words into the most likely sequence of words using a Language Model trained on a large amount of text (helps to choose between ambiguity like “WRECK A NICE BEACH” vs “RECOGNIZE SPEECH”).

An End-to-End speech recognition model is fairly straightforward by comparison. Rather than a series of small models, the bulk of the transcript is performed by one large neural network model. Depending on the specific E2E architecture, there may be a small amount of light feature generation on the input side of E2E the neural network and a little bit of processing on the output side to put together the transcript, but the decoding process is much more straightforward than the hybrid approach and most of the word is performed in one large neural network.

Selecting between Hybrid and E2E Model Types

Advantages of E2E (gen-2) models:

  • High Accuracy - Our E2E models push the state-of-the art in speech recognition accuracy in a variety of diverse use cases, and typically produce 30-50% fewer errors than gen-1 models. You can take a look at the word error rates of both hybrid and E2E models on several industry-standard test datasets here.
  • Sample Rate Flexibility - All of our E2E models can transcribe both 8khz telephone audio and 16khz audio without any loss in accuracy.
  • Out-of-Vocabulary Word Support - Even words never seen during training can very often be recognized correctly.
  • Parallel Processing - The transcription of a single audio file can be easily run in parallel across multiple CPUs or on a GPU.
  • Easier Training - training models is more straightforward (if adapting or fully re-training a model to be optimized for a particular use case).
  • Low Resource Language Support - Less information is required about a new language to train a model for a new language. The phones/sounds, pronunciations of words, and vocabulary are not required. Usually, much less training data is also be needed to produce a suitable recognition model.

Advantages of Hybrid (gen-1) models:

  • Low Latency - Our hybrid models can achieve very low latency (<100ms). E2E models can be run with settings that reduce latency, but for those models there is an impact to Word-Error-Rate and the latency will not be as low as the hybrid model.
  • Efficient Transcription - Each CPU core can transcribe several audio streams at the same time.
  • Easy Customization - New words and/or pronunciations can be added at transcription time. We also offer tools that allow you to re-build models with your own text-only data that allows the Hybrid model to be more accurate on a target domain.
  • More suitable for Embedded Devices - Hybrid models can be trained to have relatively light CPU/memory/storage requirements if it needs to run on an embedded device
  • Constrained Use Cases - If transcription is being deployed in a use case that has a limited vocabulary and/or grammar (not general transcription), the hybrid model can be trained or adapted to target this use case and achieve extremely high accuracy. Examples would be voice command-and-control of a device, or users speaking from a list of commands. Multiple grammars can even be supported and swapped in/out of the recognizer when it is running.
  • Confidence - Per-word confidence estimates are more accurate for Hybrid models than E2E models.
  • Alternate Words - Hybrid models can return rich results beyond a 1-best transcript that contains potential alternate words/sentences for the transcribed audio.
  • Less Compute Required for Training - Training and adapting speech models requires fewer GPU/compute resources.

End-to-End models are likely to be the best choice for customers that are primarily concerned with maximizing accuracy for general transcription. However, the hybrid models may be more appropriate and even more accurate under some conditions: very low latency streaming, low compute/memory embedded transcription, highly custom/unique vocabulary, a very narrow domain (ex: speaking a small number of device directed commands), vocabulary or expected command sets that can change often (even between each audio stream passed as input). By supporting both types of models and offering several different options for model customization, Cobalt is able to satisfy nearly any use case that a customer may require. The support of multiple model types also future-proofs the service by ensuring that if an improved type of speech recognition model becomes available in the future, users of cobalt-transcribe will be able to start using it with minimal changes to their API.

Our gen-2 E2E models currently do not support word level confidence, confusion networks outputs, recognition context, or GPU support during decoding. However, these features will be added to the E2E models soon.

1.1.9 - API Reference

Detailed reference for API requests and types.

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

TranscribeService

Service that implements the Cobalt Transcribe Speech Recognition API.

Version

Version(VersionRequest) VersionResponse

Queries the version of the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

Retrieves a list of available speech recognition models.

StreamingRecognize

StreamingRecognize(StreamingRecognizeRequest) StreamingRecognizeResponse

Performs bidirectional streaming speech recognition. Receive results while sending audio. This method is only available via GRPC and not via HTTP+JSON. However, a web browser may use websockets to use this service.

CompileContext

CompileContext(CompileContextRequest) CompileContextResponse

Compiles recognition context information, such as a specialized list of words or phrases, into a compact, efficient form to send with subsequent StreamingRecognize requests to customize speech recognition. For example, a list of contact names may be compiled in a mobile app and sent with each recognition request so that the app user’s contact names are more likely to be recognized than arbitrary names. This pre-compilation ensures that there is no added latency for the recognition request. It is important to note that in order to compile context for a model, that model has to support context in the first place, which can be verified by checking its ModelAttributes.ContextInfo obtained via the ListModels method. Also, the compiled data will be model specific; that is, the data compiled for one model will generally not be usable with a different model.

Messages

  • If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
  • If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

AudioFormatRAW

Details of audio in raw format

Fields

  • encoding (AudioEncoding ) Encoding of the samples. It must be specified explicitly and using the default value of AUDIO_ENCODING_UNSPECIFIED will result in an error.

  • bit_depth (uint32 ) Bit depth of each sample (e.g. 8, 16, 24, 32, etc.). This is a required field.

  • byte_order (ByteOrder ) Byte order of the samples. This field must be set to a value other than BYTE_ORDER_UNSPECIFIED when the bit_depth is greater than 8.

  • sample_rate (uint32 ) Sampling rate in Hz. This is a required field.

  • channels (uint32 ) Number of channels present in the audio. E.g.: 1 (mono), 2 (stereo), etc. This is a required field.

CompileContextRequest

The top-level message sent by the client for the CompileContext request. It contains a list of phrases or words, paired with a context token included in the model being used. The token specifies a category such as “menu_item”, “airport”, “contact”, “product_name” etc. The context token is used to determine the places in the recognition output where the provided list of phrases or words may appear. The allowed context tokens for a given model can be found in its ModelAttributes.ContextInfo obtained via the ListModels method.

Fields

  • model_id (string ) Unique identifier of the model to compile the context information for. The model chosen needs to support context which can be verified by checking its ModelAttributes.ContextInfo obtained via ListModels.

  • token (string ) The token that is associated with the provided list of phrases or words (e.g “menu_item”, “airport” etc.). Must be one of the tokens included in the model being used, which can be retrieved by calling the ListModels method.

  • phrases (ContextPhrase repeated) List of phrases and/or words to be compiled.

CompileContextResponse

The message returned to the client by the CompileContext method.

Fields

  • context (CompiledContext ) Context information in a compact form that is efficient for use in subsequent recognition requests. The size of the compiled form will depend on the amount of text that was sent for compilation. For 1000 words it’s generally less than 100 kilobytes.

CompiledContext

Context information in a compact form that is efficient for use in subsequent recognition requests. The size of the compiled form will depend on the amount of text that was sent for compilation. For 1000 words it’s generally less than 100 kilobytes.

Fields

  • data (bytes ) The context information compiled by the CompileContext method.

ConfusionNetworkArc

An Arc inside a Confusion Network Link

Fields

  • word (string ) Word in the recognized transcript

  • confidence (double ) Confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized.

  • features (ConfusionNetworkArcFeatures ) Features related to this arc

ConfusionNetworkArcFeatures

Features related to confusion network arcs

Fields

ConfusionNetworkArcFeatures.ConfidenceEntry

Fields

A Link inside a confusion network

Fields

  • start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio received by the recognizer and corresponding to the start of this link

  • duration_ms (uint64 ) Duration in milliseconds of the current link in the confusion network

  • arcs (ConfusionNetworkArc repeated) Arcs between this link

ContextInfo

Model information specifc to supporting recognition context.

Fields

  • supports_context (bool ) If this is set to true, the model supports taking context information into account to aid speech recognition. The information may be sent with with recognition requests via RecognitionContext inside RecognitionConfig.

  • allowed_context_tokens (string repeated) A list of tokens (e.g “name”, “airport” etc.) that serve has placeholders in the model where a client provided list of phrases or words may be used to aid speech recognition and produce the exact desired recognition output.

ContextPhrase

A phrase or word that is to be compiled into context information that can be later used to improve speech recognition during a StreamingRecognize call. Along with the phrase or word itself, there is an optional boost parameter that can be used to boost the likelihood of the phrase or word in the recognition output.

Fields

  • text (string ) The actual phrase or word.

  • boost (float ) This is an optional field. The boost factor is a positive number which is used to multiply the probability of the phrase or word appearing in the output. This setting can be used to differentiate between similar sounding words, with the desired word given a bigger boost factor.

    By default, all phrases or words provided in the RecongitionContext are given an equal probability of occurring. Boost factors larger than 1 make the phrase or word more probable and boost factors less than 1 make it less likely. A boost factor of 2 corresponds to making the phrase or word twice as likely, while a boost factor of 0.5 means half as likely.

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

  • models (Model repeated) List of models available for use.

Model

Description of a Transcribe Model

Fields

  • id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for recognition, and is specified in the RecognitionConfig message.

  • name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their recognition task.

  • attributes (ModelAttributes ) Model attributes

ModelAttributes

Attributes of a Transcribe Model

Fields

  • sample_rate (uint32 ) Audio sample rate(native) supported by the model

  • context_info (ContextInfo ) Attributes specifc to supporting recognition context.

  • supported_sample_rates (uint32 repeated) Audio sample rates other than the model’s native sample rate(sample_rate) that can be accommodated

RecognitionAlternative

A recognition hypothesis

Fields

  • transcript_formatted (string ) Text representing the transcription of the words that the user spoke.

    The transcript will be formatted according to the servers formatting configuration. If you want the raw transcript, please see the field transcript_raw. If the server is configured to not use any formatting, then this field will contain the raw transcript.

    As an example, if the spoken utterance was “four people”, and the server was configured to format numbers, this field would be set to “4 people”.

  • transcript_raw (string ) Text representing the transcription of the words that the user spoke, without any formatting applied. If you want the formatted transcript, please see the field transcript_formatted.

    As an example, if the spoken utterance was four people, this field would be set to “FOUR PEOPLE”.

  • start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio received by the recognizer and corresponding to the start of this utterance.

  • duration_ms (uint64 ) Duration in milliseconds of the current utterance in the spoken audio.

  • confidence (double ) Confidence estimate between 0 and 1. A higher number represents a higher likelihood of the output being correct.

  • word_details (WordDetails ) Word-level details corresponding to the transcripts. This is available only if enable_word_details was set to true in the RecognitionConfig.

RecognitionAudio

Audio to be sent to the recognizer

Fields

RecognitionConfig

Configuration for setting up a Recognizer

Fields

  • model_id (string ) Unique identifier of the model to use, as obtained from a Model message.

  • oneof audio_format.audio_format_raw (AudioFormatRAW ) Audio is raw data without any headers

  • oneof audio_format.audio_format_headered (AudioFormatHeadered ) Audio has a self-describing header. Headers are expected to be sent at the beginning of the entire audio file/stream, and not in every RecognitionAudio message.

    The default value of this type is AUDIO_FORMAT_HEADERED_UNSPECIFIED. If this value is used, the server may attempt to detect the format of the audio. However, it is recommended that the exact format be specified.

  • selected_audio_channels (uint32 repeated) This is an optional field. If the audio has multiple channels, this field can be configured with the list of channel indices that should be considered for the recognition task. These channels are 0-indexed.

    Example: [0] for a mono file, [0, 1] for a stereo file. Example: [1] to only transcribe the second channel of a stereo file.

    If this field is not set, all the channels in the audio will be processed.

    Channels that are present in the audio may be omitted, but it is an error to include a channel index in this field that is not present in the audio. Channels may be listed in any order but the same index may not be repeated in this list.

    BAD: [0, 2] for a stereo file; BAD: [0, 0] for a mono file.

  • audio_time_offset_ms (uint64 ) This is an optional field. It can be used to indicate that the audio being streamed to the recognizer is offset from the original stream by the provided duration in milliseconds. This offset will be added to all timestamps in results returned by the recognizer.

    The default value of this field is 0ms, so the timestamps in the recognition result will not be modified.

    Example use case where this field can be helpful: if a recognition session was interrupted and audio needs to be sent to a new session from the point where the session was previously interrupted, the offset could be set to the point where the interruption had happened.

  • enable_word_details (bool ) This is an optional field. If this is set to true, each result will include word level details of the transcript. These details are specified in the WordDetails message. If set to false, no word-level details will be returned. The default is false.

  • enable_confusion_network (bool ) This is an optional field. If this is set to true, each result will include a confusion network. If set to false, no confusion network will be returned. The default is false. If the model being used does not support returning a confusion network, this field will have no effect. Tokens in the confusion network always correspond to tokens in the transcript_raw returned.

  • metadata (RecognitionMetadata ) This is an optional field. If there is any metadata associated with the audio being sent, use this field to provide it to the recognizer. The server may record this metadata when processing the request. The server does not use this field for any other purpose.

  • context (RecognitionContext ) This is an optional field for providing any additional context information that may aid speech recognition. This can also be used to add out-of-vocabulary words to the model or boost recognition of specific proper names or commands. Context information must be pre-compiled via the CompileContext() method.

RecognitionConfusionNetwork

Confusion network in recognition output

Fields

RecognitionContext

A collection of additional context information that may aid speech recognition. This can be used to add out-of-vocabulary words to the model or to boost recognition of specific proper names or commands.

Fields

  • compiled (CompiledContext repeated) List of compiled context information, with each entry being compiled from a list of words or phrases using the CompileContext method.

RecognitionError

Developer-facing error message about a non-fatal recognition issue.

Fields

RecognitionMetadata

Metadata associated with the audio to be recognized.

Fields

  • custom_metadata (string ) Any custom metadata that the client wants to associate with the recording. This could be a simple string (e.g. a tracing ID) or structured data (e.g. JSON).

  • custom_id (string ) This is an optional field to specify custom ID to identify the recognition session. The custom ID must be a string of upto 64 bytes, and only alphabets, digits, hyphens and underscores are allowed. This ID may be recorded by the server in logs or other storage, and should therefore not include any sensitive information.

RecognitionResult

A recognition result corresponding to a portion of audio.

Fields

  • alternatives (RecognitionAlternative repeated) An n-best list of recognition hypotheses alternatives

  • is_partial (bool ) If this is set to true, it denotes that the result is an interim partial result, and could change after more audio is processed. If unset, or set to false, it denotes that this is a final result and will not change.

    Servers are not required to implement support for returning partial results, and clients should generally not depend on their availability.

  • cnet (RecognitionConfusionNetwork ) If enable_confusion_network was set to true in the RecognitionConfig, and if the model supports it, a confusion network will be available in the results.

  • audio_channel (uint32 ) Channel of the audio file that this result was transcribed from. Channels are 0-indexed, so the for mono audio data, this value will always be 0.

StreamingRecognizeRequest

The top-level messages sent by the client for the StreamingRecognize method. In this streaming call, multiple StreamingRecognizeRequest messages should be sent. The first message must contain a RecognitionConfig message only, and all subsequent messages must contain RecognitionAudio only. All RecognitionAudio messages must contain non-empty audio. If audio content is empty, the server may choose to interpret it as end of stream and stop accepting any further messages.

Fields

StreamingRecognizeResponse

The messages returned by the server for the StreamingRecognize request. Multiple messages of this type will be delivered on the stream, for multiple results, as soon as results are available from the audio submitted so far. If the audio has multiple channels, the results of all channels will be interleaved. Results of each individual channel will be chronological. However, there is no guarantee of the order of results across channels.

Clients should process both the result and error fields in each message. At least one of these fields will be present in the message. If both result and error are present, the result is still valid.

Fields

  • result (RecognitionResult ) A new recognition result. This field will be unset if a new result is not yet available.

  • error (RecognitionError ) A non-fatal error message. If a server encountered a non-fatal error when processing the recognition request, it will be returned in this message. The server will continue to process audio and produce further results. Clients can continue streaming audio even after receiving these messages. This error message is meant to be informational.

    An example of when these errors maybe produced: audio is sampled at a lower rate than expected by model, producing possibly less accurate results.

    This field will be unset if there is no error to report.

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The message sent by the server for the Version method.

Fields

  • version (string ) Version of the server handling these requests.

WordDetails

Fields

  • formatted (WordInfo repeated) Word-level information corresponding to the transcript_formatted field.

  • raw (WordInfo repeated) Word-level information corresponding to the transcript_raw field.

WordInfo

Word level details for recognized words in a transcript

Fields

  • word (string ) The actual word in the text

  • confidence (double ) Confidence estimate between 0 and 1. A higher number represents a higher likelihood that the word was correctly recognized.

  • start_time_ms (uint64 ) Time offset in milliseconds relative to the beginning of audio received by the recognizer and corresponding to the start of this spoken word.

  • duration_ms (uint64 ) Duration in milliseconds of the current word in the spoken audio.

Enums

AudioEncoding

The encoding of the audio data to be sent for recognition.

NameNumberDescription
AUDIO_ENCODING_UNSPECIFIED0AUDIO_ENCODING_UNSPECIFIED is the default value of this type and will result in an error.
AUDIO_ENCODING_SIGNED1PCM signed-integer
AUDIO_ENCODING_UNSIGNED2PCM unsigned-integer
AUDIO_ENCODING_IEEE_FLOAT3PCM IEEE-Float
AUDIO_ENCODING_ULAW4G.711 mu-law
AUDIO_ENCODING_ALAW5G.711 a-law

AudioFormatHeadered

NameNumberDescription
AUDIO_FORMAT_HEADERED_UNSPECIFIED0AUDIO_FORMAT_HEADERED_UNSPECIFIED is the default value of this type.
AUDIO_FORMAT_HEADERED_WAV1WAV with RIFF headers
AUDIO_FORMAT_HEADERED_MP32MP3 format with a valid frame header at the beginning of data
AUDIO_FORMAT_HEADERED_FLAC3FLAC format
AUDIO_FORMAT_HEADERED_OGG_OPUS4Opus format with OGG header

ByteOrder

Byte order of multi-byte data

NameNumberDescription
BYTE_ORDER_UNSPECIFIED0BYTE_ORDER_UNSPECIFIED is the default value of this type.
BYTE_ORDER_LITTLE_ENDIAN1Little Endian byte order
BYTE_ORDER_BIG_ENDIAN2Big Endian byte order

Scalar Value Types

.proto TypeC++ TypeC# TypeGo TypeJava TypePHP TypePython TypeRuby Type

double
doubledoublefloat64doublefloatfloatFloat

float
floatfloatfloat32floatfloatfloatFloat

int32
int32intint32intintegerintBignum or Fixnum (as required)

int64
int64longint64longinteger/stringint/longBignum

uint32
uint32uintuint32intintegerint/longBignum or Fixnum (as required)

uint64
uint64ulonguint64longinteger/stringint/longBignum or Fixnum (as required)

sint32
int32intint32intintegerintBignum or Fixnum (as required)

sint64
int64longint64longinteger/stringint/longBignum

fixed32
uint32uintuint32intintegerintBignum or Fixnum (as required)

fixed64
uint64ulonguint64longinteger/stringint/longBignum

sfixed32
int32intint32intintegerintBignum or Fixnum (as required)

sfixed64
int64longint64longinteger/stringint/longBignum

bool
boolboolboolbooleanbooleanbooleanTrueClass/FalseClass

string
stringstringstringStringstringstr/unicodeString (UTF-8)

bytes
stringByteString[]byteByteStringstringstrString (ASCII-8BIT)

1.1.10 - FAQ

Answers to Frequently Asked Questions.

System Requirements

Does Cobalt Transcribe run on Linux?

Yes, you can run Cobalt Transcribe on Linux natively or via Docker. Check out the documentation to get started.

Does Cobalt Transcribe run on macOS?

Yes, you can run Cobalt Transcribe on macOS via Docker Desktop for evaluation purpose. Check out the documentation to get started. However, we don’t recommend running Cobalt Transcribe on macOS in production.

Does Cobalt Transcribe run on Windows?

Yes, Windows is supported via Docker Desktop for evaluation. We don’t recommend running Cobalt Transcribe on Windows in production.

Does Cobalt Transcribe run on embedded devices?

Yes, Cobalt Transcribe supports embedded devices such as Raspberry Pi, Tegra etc. However, you’ll probably want to contact us for a smaller model due to memory limitations.

Does Cobalt Transcribe run on Android or iOS?

Android and iOS require a specific implementation strategy. Please contact us for support working with Android or iOS.

What are the technical requirements for a scaled on-premise deployment?

Each containerized instance of Cobalt Transcribe should be provided with 4 cores and 8 GB RAM when using for streaming recognition.

Product Features

Which languages does Cobalt Transcribe support?

Cobalt offers speech recognition in English (US & UK), Spanish, French, German, Russian, Brazilian Portuguese, Korean, Japanese, Swahili, Cambodian. Please contact sales@cobaltspeech.com to learn more. Cobalt is always looking for partners to develop, sell, and/or market speech technology in other languages.

Can I use Cobalt Transcribe in the field of telephony such as contact centers?

Yes, Cobalt Transcribe uses low latency and/or 8kHz telephony models for transcribing telephone calls and contact center conversations. Additional insight is retrivable through high-precision timestamps and n-best transcripts. Cobalt technology provides solutions for contact centers including summarization, redaction, and sentiment analysis of conversations.

Can I redact Personal Identifiable Inforamtion (PII) from the output transcript?

PII redaction is a separate service that can be integrated with Cobalt Transcribe. Please contact us for details.

Does Cobalt Transcribe support real-time transcription?

Yes, Cobalt Transcribe can accept audio samples as they are recorded and will provide streaming output with relatively low latency. It also supports output of partial results, which is available almost immediately during decoding. This feature is useful in real-time interfaces where users can see what is being recognized nearly immediately as they speak, but some words in the preliminary output may be corrected in a final result as more audio and context is available. Cobalt transcribe performs automatic end-pointing to determine the end of utterance.

Recognition Accuracy

How accurate is Cobalt Transcribe?

Cobalt Transcribe is available in two different architectures: Hybrid and End-to-End. We have evaluated the word error rate (WER) of both versions of Cobalt Transcribe on several industry-standard test datasets:

DatasetDomainHybrid WEREnd-to-End WER
CommonVoice-testRead Speech, Crowd Sourced11.5%5.0%
Librispeech-testRead Speech, Audiobooks, Crowd Sourced6.2%2.2%
Tedlium-testSpontaneous Speech, Presentations7.5%3.9%
WallStreetJournal-testRead Speech, News7.4%5.8%
MultilingualLibriSpeech-testRead Speech, Audiobooks, Crowd Sourced8.8%4.0%
OHSU-testSpontaneous Speech, Children’s Speech16.9%12.4%

The WER is dependent on a number of factors such as the train-test split, formatting of the decoded transcript, accuracy vs. latency trade-offs, etc. Therefore, these numbers are not directly comparable to the WERs reported by other service providers, even on the same dataset.

How do I further improve audio transcription accuracy?

Our base models are trained on a large amount of audio and text to ensure robust accuracy on a variety of use cases. The configurable nature of Cobalt Transcribe’s models allows for updates that can improve transcription accuracy specific to your use-case:

  • Adding vocabulary and context via the RecognitionContext API: This will help you to capture proper names and domain-specific terminology correctly.

  • End-to-end (E2E) models typically have better accuracy and more robust recognition performance for different accents and dialects. However, E2E models are more computationally expensive, and tend to have higher latency when compared to hybrid models. If you would like to try one out please contact us at sales@cobaltspeech.com

  • Continuous adaptation of acoustic models (AMs) using Cobalt Transcribe Tuner. This continuous learning framework automatically updates the acoustic model using your production data. For more information, contact sales@cobaltspeech.com.

  • Cobalt’s speech scientists can work with you to optimize accuracy for your conditions and application: For speech recognition in a specific acoustic environment or domain-specific use case (e.g. noisy factory floor, airport, surgical lab, patient-doctor conversations, quarterly earnings calls, etc.) we can adapt the acoustic and language models using relevant audio and text data.

I am starting a new speech project. How can I get the best transcription accuracy?

The transcription accuracy depends on several factors such as:

  • Appropriate sampling rate (8kHz / 16kHz) and matching model
  • Audio format : Lossless codecs like wav or flac may be better over mp3 or ogg
  • Microphone selection, placement, directionality (Cardiod, Omni)
  • Trade-offs between latency and accuracy.
  • Consider constrained grammar or providing recognition context
  • AM and LM adaptation

Recognition Speed/Performance

How do I further improve audio transcription latency?

One way to improve the latency is to make the streaming buffer size smaller. We recommend setting the streaming buffer size between 512 bytes and 4096 bytes. We can also work with you to tune model parameters such as beam search width to reduce the latency. Our speech scientists can also make a smaller model for your application, or tune parameters and model size for optimal latency and accuracy trade-offs. If you’re interested in this, contact sales@cobaltspeech.com.

How long does it take to transcribe audio?

The processing speed of speech to text conversion is measured by the real time factor (RTF) which is the ratio of time taken to transcribe an audio file to the duration of the audio. Cobalt Transcribe has an RTF of 0.16 and 0.4 using our general purpose hybrid and E2E models, respectively. That means transcribing one hour of speech typically takes approximately 10 minutes for the hybrid model and about 24 minutes for the E2E model.

Are there limits on the number of jobs that can be processed concurrently?

Number of concurrent audio channels depends on the models being used and the CPU. Our general purpose models typically support 6 channels per core for realtime streams when running on a CPU such as C6i EC2 instances.

What does it cost, in terms of CPU resources, to transcribe a million minutes of speech?

With our standard models, Cobalt Transcribe can run 6 channels per core for realtime speech input, assuming a c6i EC2 processor. A 4-core processor can therefore transcribe 24 minutes of audio per minute of wall-clock time. At current AWS pricing, an EC2 instance costs $0.17 per hour. Therefore, the cost for a million minutes of speech is approximately $120.

Costs can be significantly lower when using c7g instances on EC2. Contact us for more information.

How scalable is Cobalt Transcribe? How can I carry out large deployments?

Cobalt Transcribe can scale between large-scale servers and low-power embedded hardware. For large-scale deployments, you can increase the number of concurrent audio channels for faster decoding. Cobalt Transcribe has the capability to decode using separate threads. Our general purpose models typically support 6 channels per core when running on a CPU such as C6i EC2 instances. Moreover, you can deploy Cobalt Transcribe via docker and kubernetes to automatically scale up (or down) your resources according to your demand in a cost effective manner without causing a decline in performance.

Recognition accuracy vs speed/performance

How do I choose between hybrid and end-to-end models?

End-to-End models are likely to be the best choice for customers that are primarily concerned with maximizing accuracy for general transcription. However, the hybrid models may be more appropriate and even more accurate under some conditions: very low latency streaming, low compute/memory embedded transcription, highly custom/unique vocabulary, a very narrow domain (ex: speaking a small number of device directed commands), vocabulary or expected command sets that can change often (even between each audio stream passed as input). For detailed comparison between hybrid and end-to-end models, you may take a look at Hybrid vs End-to-End Models.

Supported Audio Formats

What type of media files does Cobalt Transcribe support?

Cobalt Transcribe supports common media formats, such as WAV, MP3, FLAC, Ogg and audio encoding like PCM, mu-law, and a-law. Raw audio format is also supported.

My audio source is 48kHz/44.1kHz. Does Cobalt Transcribe support that?

Yes, we resample the audio to an appropriate sampling rate automatically. Please note that our default sampling rates are 16kHz for wideband models and 8kHz for telephony models. Accuracy improvements for higher sampling rates (than 16kHz) are minimal, and not generally worth the associated increase in data rates and data transfer requirements, or the additional overhead for resampling.

API and Integration

How do I test Cobalt Transcribe?

We are happy to offer free trials under a software evaluation license of Cobalt Transcribe with all available features. Typically, our software evaluation licenses are for a period of 30 days. To get started with Cobalt Transcribe, please check the quick start.

Please try our Cobalt Transcribe Speech Recognition Demo for simple evaluation purposes. This demo server is for testing and demonstration purposes only and is not guaranteed to support high availability or high volume.

Which SDKs are available to integrate Cobalt Transcribe into my project?

Cobalt Transcribe uses gRPC to define its APIs. The API is defined as a protobuf schema, and grpc tools can be used to generate client SDKs in several languages, including Python, Go, C++, Java, C#, etc.

Can I use my own models with Cobalt Transcribe?

Cobalt Transcribe models are trained on thousands of hours of data and produce very accurate transcripts over a wide range of different conditions. We provide tools and services that allow our models to be tailored towards your particular use case if additional accuracy is desired. If customers have their own existing Kaldi or wav2vec 2.0 models, Cobalt Transcribe supports the use of those external models.

Product Comparison

What are the benefits of Cobalt Transcribe over other speech-to-text services?

Compared to other services, Cobalt Transcribe offers the following advantages:

  • You can host the Cobalt Transcribe server on your system locally or in your virtual private cloud. This enables you to keep your data private and secure.
  • Cobalt Transcribe has low latency. It is particularly useful for embedded devices and real-time applications.
  • Cobalt Transcribe is highly customizable. Adaptaing the language and acoustic models to your specific terminology will improve performance.
  • Cobalt’s experienced speech scientists are available to adapt the LM and AM to your target domain for the best recognition results.
  • The Cobalt Transcribe API supports several outputs: 1-best results, per-word start times and durations, per-word confidences, n-best transcripts, confusion networks, and lattices.

2 - Voice Intelligence

2.1 - VoiceBio

Low latency, high accuracy on-prem / on-cloud solutions for speaker verification and identification.

VoiceBio turns speech into a voiceprint — a compact representation of what a speaker’s voice sounds like, independent of what they said. Once you hold a voiceprint you can answer two questions:

  • Is this the person they claim to be? StreamingVerify compares live audio against one voiceprint and returns a similarity score and a match decision.
  • Which of these people is speaking? StreamingIdentify compares live audio against a list of voiceprints and returns the best match, or none.

Voiceprints come from StreamingEnroll, which reports how much more audio it needs before enrollment is complete. They can be stored and reused: to compare voiceprints you already hold, without sending audio again, use CompareVoiceprints, and to store them as numerical vectors for your own indexing, use VectorizeVoiceprints.

The engine runs on your own hardware, so the audio and the voiceprints stay inside your infrastructure.

2.1.1 - Getting Started

How to get a VoiceBio Server running on your system

Using Cobalt VoiceBio

  • A typical VoiceBio release, provided as a compressed archive, will contain a linux binary (voicebio-server) for the required native CPU architecture, appropriate Dockerfile and models.

  • Cobalt VoiceBio runs either locally on linux or using Docker.

  • Cobalt VoiceBio will serve the VoiceBio GRPC API on port 2727.

  • To quickly try out VoiceBio, first start the server as shown below and use the SDK in your preferred language to use VoiceBio from the command line or within your application.

Running VoiceBio Server Locally on Linux

./voicebio-server
  • By default, the binary assumes the presence of a configuration file, located in the same directory, named: voicebio-server.cfg.toml. A different config file may be specified using the --config argument.

Running VoiceBio Server as a Docker Container

To build and run the Docker image for VoiceBio, run:

docker build -t cobalt-voicebio .
docker run -p 2727:2727 -p 8080:8080 cobalt-voicebio

How to Get a Copy of the VoiceBio Server and Models

Contact us for getting a release best suited to your requirements.

The release you will receive is a compressed archive (tar.bz2) and is generally structured accordingly:

release.tar.bz2
├── COPYING
├── README.md
├── voicebio-server
├── voicebio-server.cfg.toml
├── Dockerfile
├── models
│   └── en_US-16khz
│
└── cobalt.license.key [ provided separately, needs to be copied over ]
  • The README.md file contains information about this release and instructions for how to start the server on your system.

  • The voicebio-server is the server program which is configured using the voicebio-server.cfg.toml file.

  • The Dockerfile can be used to create a container that will let you run VoiceBio server on non-linux systems such as MacOS and Windows.

  • The models directory contains the speaker ID models. The content of these directory will depend on the models you are provided.

System Requirements

Cobalt VoiceBio runs on Linux. You can run it directly as a linux application.

You can evaluate the product on Windows or Linux using Docker Desktop but we would not recommend this setup for use in a production environment.

A Cobalt VoiceBio release typically includes a single VoiceBio model together with binaries and config files. The general purpose VoiceBio models take up to 100MB of disk space, and need a minimum of 2GB RAM when evaluating locally. For production workloads, we recommend configuring containerized applications with each instance allocated with 4 CPUs and 4GB RAM.

Cobalt VoiceBio runs on x86_64 CPUs. We also support Arm64 CPUs, including processors such as the Graviton (AWS c7g EC2 instances). VoiceBio is significantly more cost effective to run on C7g instances compared to similarly sized Intel or AMD processors, and we can provide you an Arm64 release on request.

To integrate Cobalt VoiceBio into your application, please follow the next steps to install or generate the SDK in a language of your choice.

2.1.2 - Generating SDKs

Gives instructions about how to generate an SDK for your project from the proto API definition.
  • APIs for all Cobalt’s services are defined as a protocol buffer specification or simply a proto file and be found in the cobaltspeech/proto github repository.

  • The proto file allows a developer to auto-generate client SDKs for a number of different programming languages. Step by step instructions for generating your own SDK can be found below.

  • We provide pre-generated SDKs for a couple of languages. You can choose to use these instead of generating your own. These are listed here along with instructions on how to install / import them into your projects.

Pre-generated SDKs

Golang

import voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
  • An example client using the above repo can be found here.

Python

  • Pre-generated SDK files for Python can be found in the cobaltspeech/py-genproto repo

  • The Python SDK depends on Python >= 3.5. You may use pip to perform a system-wide install, or use virtualenv for a local install. To use it in your Python project, install it:

pip install --upgrade pip
pip install "git+https://github.com/cobaltspeech/py-genproto"

Generating SDKs

Step 1. Installing buf

  • To work with proto files, we recommend using buf, a user-friendly command line tool that can be configured generate documentation, schemas and SDK code for different languages.
# Latest version as of March 14th, 2023.

COBALT="${HOME}/cobalt"
  mkdir -p "${COBALT}/bin"

VERSION="1.15.1"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
  curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and adding to $PATH.

chmod +x "${COBALT}/bin/buf"
  export PATH="${PATH}:${COBALT}/bin"
brew install bufbuild/buf/buf

Step 2. Getting proto files

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Change this to where you want to clone the repo to.
PROTO_REPO="${COBALT}/git/proto"

git clone https://github.com/cobaltspeech/proto "${PROTO_REPO}"

Step 3. Generating code

  • The cobaltspeech/proto repo provides a buf.gen.yaml config file to get you started with a couple of languages.

  • Other plugins can be added to the buf.gen.yaml file to generate SDK code for more languages.

  • To generate the SDKs, simply run the following (assuming the buf binary is in your $PATH)

cd "${PROTO_REPO}"

# Removing any previously generated files.
rm -rf ./gen

# Generating code for all proto files inside the `proto` directory.
buf generate proto
  • You should now have a folder called gen inside ${PROTO_REPO} that contains the generated code. The latest version of the VoiceBio API is v1. You can import / include / copy the generated files into your projects as per the conventions of different languages.
gen
├── ... other languages ...
└── py
  └── cobaltspeech
    ├── ... other services ...
    └── voicebio
      └── v1
        ├── voicebio_pb2_grpc.py
        ├── voicebio_pb2.py
        └── voicebio_pb2.pyi
gen
├── ... other languages ...
└── go
   ├── cobaltspeech
   │ ├── ...
   │   └── voicebio
   │      └── v1
   │        ├── voicebio_grpc.pb.go
   │        └── voicebio.pb.go
   └── gw
     └── cobaltspeech
       ├── ...
       └── voicebio
         └── v1
            └── voicebio.pb.gw.go

Step 4. Installing gPRC and protobuf

  • A couple of gRPC and protobuf dependencies are required along with the code generated above. The method of installing them depends on the programming language being used.
  • These dependencies and the most common way of installing/ / including them are listed below for some chosen languages.
# It is encouraged to this inside a python virtual environment

# to avoid creating version conflicts for other scripts that may

# be using these libraries.

pip install --upgrade protobuf
pip install --upgrade grpcio
pip install --upgrade google-api-python-client
go get google.golang.org/protobuf
go get google.golang.org/grpc
go get google.golang.org/genproto
# More details on grpc installation can be found at:

# https://grpc.io/docs/languages/cpp/quickstart/

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Latest version as of 14th March, 2023.

VERSION="v1.52.0"
GRPC_REPO="${COBALT}/git/grpc-${VERSION}"

git clone \
 --recurse-submodules --depth 1 --shallow-submodules \
 -b "${VERSION}" \
 https://github.com/grpc/grpc ${GRPC_REPO}

cd "${GRPC_REPO}"
mkdir -p cmake/build

# Change this to where you want to install libprotobuf and libgrpc.

# It is encouraged to install gRPC locally as there is no easy way to

# uninstall gRPC after you’ve installed it globally.

INSTALL_DIR="${COBALT}"

cd cmake/build
cmake \
 -DgRPC_INSTALL=ON \
 -DgRPC_BUILD_TESTS=OFF \
 -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
 ../..

make -j
make install

2.1.3 - Connecting to the Server

Describes how to connect to a running Cobalt VoiceBio server instance.
  • Once you have your VoiceBio server up and running, and have installed or generated the SDK for your project, you can connect to a running instance of VoiceBio server, by “dialing” a gRPC connection.

  • First, you need to know the address where the server is running: e.g. host:grpc_port. By default, this is localhost:2727 and should be logged to the terminal when you first start VoiceBio server as grpcAddr:

2023/08/14 10:49:38 info  {"license":"Copyright © 2023--present. Cobalt Speech and Language, Inc.  For additional details, including information about open source components used in this software, please see the COPYING file bundled with this program."}
2023/08/14 10:49:38 info  {"msg":"reading config file","path":"configs/voicebio-server.config.toml"}
2023/08/14 10:49:38 info  {"msg":"server initializing"}
2023/08/14 10:49:38 info  {"msg":"license verified"}
2023/08/14 10:49:41 info  {"msg":"runtime initialized","model_count":"2","init_time_taken":"2.512935646s"}
2023/08/14 10:49:41 info  {"msg":"server started","grpcAddr":"[::]:2727","httpApiAddr":"[::]:8080","httpOpsAddr":"[::]:8081"}

Default Connection

The following code snippet connects to the server and queries its version. It connects to the server using an “insecure” gRPC channel. This would be the case if you have just started up a local instance of VoiceBio server without TLS enabled.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress  = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebiopb.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebiopb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Connect with TLS

  • In our recommended setup for deployment, TLS is enabled in the gRPC connection, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

  • The following snippets show how to connect to a VoiceBio Server that has TLS enabled. They use the cobalt’s self-hosted demo server at demo.cobaltspeech.com:2727, but you obviously use your own server instance.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "demo.cobaltspeech.com:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	voicebiopb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress  = "demo.cobaltspeech.com:2727"
		connectTimeout = 10 * time.Second
	)

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout)
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(creds),
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebiopb.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebiopb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

  • In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your VoiceBio server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

  • Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS.

  • The following snippets show how to present client certificates when setting up the credentials. These could then be used in the same way as the examples above to connect to a TLS enabled server.

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string 
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)
package main

import (
	// ...

	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	// ..
)

func main() {
	// ...

	// Root PEM certificate for validating self-signed server certificate
	var rootCert []byte

	// Client PEM certificate and private key.
	var certPem, keyPem []byte

	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
		fmt.Printf("unable to use given caCert\n")
		os.Exit(1)
	}

	clientCert, err := tls.X509KeyPair(certPem, keyPem)
	if err != nil {
		fmt.Printf("unable to use given client certificate and key: %v\n", err)
		os.Exit(1)
	}

	tlsCfg := tls.Config{
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{clientCert},
	}

	creds := credentials.NewTLS(&tlsCfg)

	// ...
}

2.1.4 - Streaming Enrollment

Describes how to stream audio to VoiceBio server for enrollment.
  • The following example shows how to stream audio using VoiceBio’s StreamingEnroll request and generate a voiceprint. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

  • We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here. For best accuracy, it is recommended to use an uncompressed / loss-less compression audio format like WAV or FLAC.

  • The examples below use a WAV file as input. We will query the server for available models and use the first model to generate the voiceprint.

  • Generated Voiceprints can be updated and made more robust by re-enrolling them with additional audio. Please see the re-enrollment section.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Set the enrollment config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = voicebio.EnrollmentConfig(
    model_id=modelID,
    previous_voiceprint=None,
)

# The first request to the server should only contain the
# configuration. Subsequent requests should contain audio
# bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingEnrollRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingEnrollRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
with open("test.wav", "rb") as audio:
  result = client.StreamingEnroll(stream(cfg, audio))

# A certain minimum duration of speech is required for completing enrollment.
# The enrollment status contains information on Whether that has been met or
# whether additional audio is required.  
print(f"enrollment Status:\n{result.enrollment_status}\n")

# Saving the voiceprint data to a file. This can be provided again
# in another StreamingEnroll request (for continuing enrollment) or
# submitted for verification / identification requests.
with open("voiceprint.bin", 'w') as f:
  f.write(result.voiceprint.data)
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting the first model.
	cfg := &voicebio.EnrollmentConfig{
		ModelId:            modelResp.Models[0].Id,
		PreviousVoiceprint: nil,
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting enrollment.
	result, err := StreamingEnroll(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming enrollment: %v\n", err)
		os.Exit(1)
	}

	// A certain minimum duration of speech is required for completing enrollment.
	// The enrollment status contains information on Whether that has been met or
	// whether additional audio is required.
	fmt.Printf("Enrollment Status: %v\n", result.EnrollmentStatus)

	// Saving the voiceprint data to a file. This can be provided again
	// in another StreamingEnroll request (for continuing enrollment) or
	// submitted for verification / identification requests.
	if err := os.WriteFile("voiceprint.bin", []byte(result.Voiceprint.Data), os.ModePerm); err != nil {
		fmt.Printf("failed to write voiceprint data: %v\n", err)
		os.Exit(1)
	}
}

// StreamingEnroll wraps the streaming API for performing speaker enrollment
// (i.e. voiceprint generation) using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingEnroll(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
) (*voicebio.StreamingEnrollResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingEnroll(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingEnrollClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingEnrollRequest{
		Request: &voicebio.StreamingEnrollRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingEnrollRequest{
				Request: &voicebio.StreamingEnrollRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

Streaming from microphone

  • Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

  • The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import subprocess
import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = voicebio.EnrollmentConfig(
    model_id=modelID,
    previous_voiceprint=None,
    audio_format=voicebio.AudioFormat(
      audio_format_raw=voicebio.AudioFormatRAW(
        encoding="AUDIO_ENCODING_SIGNED",
        bit_depth=16,
        byte_order="BYTE_ORDER_LITTLE_ENDIAN",
        sample_rate=m.attributes.sample_rate,
        channels=1,
      )
    ),
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above for *10 seconds*.
maxDuration = 10
cmd = f"rec -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 - trim 0 {maxDuration}"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
    print(f"[ERROR] failed to read audio from mic stream: {err}")

print(f"\n[INFO] recording {maxDuration} seconds of audio microphone ... \n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingEnrollRequest(config=cfg)

    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingEnrollRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
result = client.StreamingEnroll(stream(cfg, audio))

# A certain minimum duration of speech is required for completing enrollment.
# The enrollment status contains information on Whether that has been met or
# whether additional audio is required.  
print(f"enrollment Status:\n{result.enrollment_status}\n")

# Saving the voiceprint data to a file. This can be provided again
# in another StreamingEnroll request (for continuing enrollment) or
# submitted for verification / identification requests.
with open("voiceprint.bin", 'w') as f:
  f.write(result.voiceprint.data)

audio.close()
mic.kill()
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"strings"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &voicebio.EnrollmentConfig{
		ModelId:            m.Id,
		PreviousVoiceprint: nil,
		AudioFormat: &voicebio.AudioFormat{AudioFormat: &voicebio.AudioFormat_AudioFormatRaw{
			AudioFormatRaw: &voicebio.AudioFormatRAW{
				Encoding:   voicebio.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  voicebio.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above for *10 seconds*.
	maxDuration := 10
	args := fmt.Sprintf("-t raw -r %d -e signed -b 16 -L -c 1 - trim 0 %d", m.Attributes.SampleRate, maxDuration)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)
	cmd.Stderr = os.Stderr

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording %d seconds from microphone \n", maxDuration)

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	// Starting enrollment.
	result, err := StreamingEnroll(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming enrollment: %v\n", err)
		os.Exit(1)
	}

	if err := eg.Wait(); err != nil {
		fmt.Printf("%v\n", err)
		os.Exit(1)
	}

	// A certain minimum duration of speech is required for completing enrollment.
	// The enrollment status contains information on Whether that has been met or
	// whether additional audio is required.
	fmt.Printf("Enrollment Status: %v\n", result.EnrollmentStatus)

	// Saving the voiceprint data to a file. This can be provided again
	// in another StreamingEnroll request (for continuing enrollment) or
	// submitted for verification / identification requests.
	if err := os.WriteFile("voiceprint.bin", []byte(result.Voiceprint.Data), os.ModePerm); err != nil {
		fmt.Printf("failed to wriet voiceprint data: %v\n", err)
		os.Exit(1)
	}
}

// StreamingEnroll wraps the streaming API for performing speaker enrollment
// (i.e. voiceprint generation) using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingEnroll(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
) (*voicebio.StreamingEnrollResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingEnroll(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingEnrollClient,
	cfg *voicebio.EnrollmentConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingEnrollRequest{
		Request: &voicebio.StreamingEnrollRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingEnrollRequest{
				Request: &voicebio.StreamingEnrollRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

Re-enrollment

  • Voiceprints can be updated and made more robust by re-enrolling them with additional audio. This can be easily done by providing previous voiceprint data in the EnrollmentConfig along with additional audio in a new StreamingEnroll request.
# Connect to server ...

with open("voiceprint.bin", 'r') as f:
  voiceprint = f.read().strip()

cfg = voicebio.EnrollmentConfig(
  model_id=modelID,
  previous_voiceprint=voicebio.Voiceprint(data=voiceprint),
)

# Send audio to server ...
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {

	// Connect to server ...

	// Reading old voiceprint data.
	data, err := os.ReadFile("voiceprint.bin")
	if err != nil {
		fmt.Printf("\nfailed to read voiceprint data: %v\n", err)
		os.Exit(1)
	}

	cfg := &voicebio.EnrollmentConfig{
		ModelId:            modelResp.Models[0].Id,
		PreviousVoiceprint: &voicebio.Voiceprint{Data: string(data)},
	}

	// Send audio to server ...
}

2.1.5 - Streaming Verification

Describes how to stream audio to VoiceBio server for verification against a voiceprint.
  • The following example shows how to stream audio using VoiceBio’s StreamingVerify request and verify whether the audio matches the provided voiceprint. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

  • We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here. For best accuracy, it is recommended to use an uncompressed / loss-less compression audio format like WAV or FLAC.

  • The examples below use a WAV file as input. We will query the server for available models and use the first model to score and verify given audio against a given voiceprint.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Loading reference voiceprint.
with open("voiceprint.bin", 'r') as f:
    voiceprint = voicebio.Voiceprint(data=f.read().strip())

# Set the verification config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = voicebio.VerificationConfig(
    model_id=modelID,
    voiceprint=voiceprint,
)

# The first request to the server should only contain the
# configuration. Subsequent requests should contain audio
# bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingVerifyRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingVerifyRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
with open("test.wav", "rb") as audio:
  resp = client.StreamingVerify(stream(cfg, audio))

# Server returns a similarity score along with whether the score
# exceeded the server-configured threshold for being a match.
print(f"Verification Score: {resp.result.similarity_score:1.3f}, Match: {resp.result.is_match}")
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Reading voiceprint data.
	data, err := os.ReadFile("voiceprint.bin")
	if err != nil {
		fmt.Printf("\nfailed to read voiceprint data: %v\n", err)
		os.Exit(1)
	}

	// Selecting the first model.
	cfg := &voicebio.VerificationConfig{
		ModelId:    modelResp.Models[0].Id,
		Voiceprint: &voicebio.Voiceprint{Data: string(data)},
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting verification.
	resp, err := StreamingVerify(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming verification: %v\n", err)
		os.Exit(1)
	}

	// Server returns a similarity score along with whether the score
	// exceeded the server-configured threshold for being a match.
	fmt.Printf("Verification Score: %1.3f, Match: %v\n", resp.Result.SimilarityScore, resp.Result.IsMatch)
}

// StreamingVerify wraps the streaming API for performing speaker verification
// using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingVerify(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.VerificationConfig,
	audio io.Reader,
) (*voicebio.StreamingVerifyResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingVerify(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingVerifyClient,
	cfg *voicebio.VerificationConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingVerifyRequest{
		Request: &voicebio.StreamingVerifyRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingVerifyRequest{
				Request: &voicebio.StreamingVerifyRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

Streaming from microphone

  • Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

  • The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import subprocess
import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Loading reference voiceprint.
with open("voiceprint.bin", 'r') as f:
    voiceprint = voicebio.Voiceprint(data=f.read().strip())

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = voicebio.VerificationConfig(
    model_id=modelID,
    voiceprint=voiceprint,
    audio_format=voicebio.AudioFormat(
      audio_format_raw=voicebio.AudioFormatRAW(
        encoding="AUDIO_ENCODING_SIGNED",
        bit_depth=16,
        byte_order="BYTE_ORDER_LITTLE_ENDIAN",
        sample_rate=m.attributes.sample_rate,
        channels=1,
      )
    ),
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above for *10 seconds*.
maxDuration = 10
cmd = f"rec -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 - trim 0 {maxDuration}"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
    print(f"[ERROR] failed to read audio from mic stream: {err}")

print(f"\n[INFO] recording {maxDuration} seconds of audio microphone ... \n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingVerifyRequest(config=cfg)

    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingVerifyRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
resp = client.StreamingVerify(stream(cfg, audio))

# Server returns a similarity score along with whether the score
# exceeded the server-configured threshold for being a match.
print(f"Verification Score: {resp.result.similarity_score:1.3f}, Match: {resp.result.is_match}")

audio.close()
mic.kill()
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"strings"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Reading voiceprint data.
	data, err := os.ReadFile("voiceprint.bin")
	if err != nil {
		fmt.Printf("\nfailed to read voiceprint data: %v\n", err)
		os.Exit(1)
	}

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &voicebio.VerificationConfig{
		ModelId:    m.Id,
		Voiceprint: &voicebio.Voiceprint{Data: string(data)},
		AudioFormat: &voicebio.AudioFormat{AudioFormat: &voicebio.AudioFormat_AudioFormatRaw{
			AudioFormatRaw: &voicebio.AudioFormatRAW{
				Encoding:   voicebio.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  voicebio.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above for *10 seconds*.
	maxDuration := 10
	args := fmt.Sprintf("-t raw -r %d -e signed -b 16 -L -c 1 - trim 0 %d", m.Attributes.SampleRate, maxDuration)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)
	cmd.Stderr = os.Stderr

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording %d seconds from microphone \n", maxDuration)

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	// Starting verification.
	resp, err := StreamingVerify(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming verification: %v\n", err)
		os.Exit(1)
	}

	// Server returns a similarity score along with whether the score
	// exceeded the server-configured threshold for being a match.
	fmt.Printf("Verification Score: %1.3f, Match: %v\n", resp.Result.SimilarityScore, resp.Result.IsMatch)
}

// StreamingVerify wraps the streaming API for performing speaker verification
// using  the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingVerify(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.VerificationConfig,
	audio io.Reader,
) (*voicebio.StreamingVerifyResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingVerify(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingVerifyClient,
	cfg *voicebio.VerificationConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingVerifyRequest{
		Request: &voicebio.StreamingVerifyRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingVerifyRequest{
				Request: &voicebio.StreamingVerifyRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

2.1.6 - Streaming Identification

Describes how to stream audio to VoiceBio server for identification using given voiceprints.
  • The following example shows how to stream audio using VoiceBio’s StreamingIdentify request and identify the speaker in the audio using provided voiceprints. The stream can come from a file on disk or be directly from a microphone in real time.

Streaming from an audio file

  • We support several headered file formats including WAV, MP3, FLAC etc. For more details, please see the protocol buffer specification here. For best accuracy, it is recommended to use an uncompressed / loss-less compression audio format like WAV or FLAC.

  • The examples below use a WAV file as input. We will query the server for available models and use the first model to score and identify the given audio against a given set of voiceprints.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
modelID = modelResp.models[0].id

# Loading reference voiceprints.
voiceprints = []
for p in ["user1.bin", "user2.bin", "user3.bin"]:
    with open(p, 'r') as f:
        voiceprints.append(voicebio.Voiceprint(data=f.read().strip()))

# Set the identification config. We don't set the audio format and let the
# server auto-detect the format from the file header.
cfg = voicebio.IdentificationConfig(
    model_id=modelID,
    voiceprints=voiceprints,
)

# The first request to the server should only contain the
# configuration. Subsequent requests should contain audio
# bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingIdentifyRequest(config=cfg)
    
    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingIdentifyRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
with open("test.wav", "rb") as audio:
  result = client.StreamingIdentify(stream(cfg, audio))

# Server returns the index of the voiceprint that matches the best, a similarity
# score for each voiceprint along with whether the score exceeded the server-configured
# threshold for being a match.
#
# If none of the voiceprints were a good match, the best match index will be negative.
matched = "❌ No Match found"
if result.best_match_index >= 0:
    best_score = result.voiceprint_comparison_results[result.best_match_index].similarity_score
    matched = f"✅ Match found: Index: {result.best_match_index}, Score: {best_score:1.3f}"

print(f"\nIdentification Result:\n")

print("Scores:")
for i, r in enumerate(result.voiceprint_comparison_results):
    print(f"Index: {i}, Score: {r.similarity_score:1.3f}, IsMatch: {r.is_match}")

print(f"\n{matched}")
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Reading voiceprint data.
	voiceprints := make([]*voicebio.Voiceprint, 0)

	for i, p := range []string{"user1.bin", "user2.bin", "user3.bin"} {
		data, err := os.ReadFile(p)
		if err != nil {
			fmt.Printf("\nfailed to read voiceprint[%d] data: %v\n", i, err)
			os.Exit(1)
		}

		voiceprints = append(voiceprints, &voicebio.Voiceprint{Data: string(data)})
	}

	// Selecting the first model.
	cfg := &voicebio.IdentificationConfig{
		ModelId:     modelResp.Models[0].Id,
		Voiceprints: voiceprints,
	}

	// Opening audio file.
	audio, err := os.Open("test.wav")
	if err != nil {
		fmt.Printf("failed to open audio file: %v\n", err)
		os.Exit(1)
	}

	defer audio.Close()

	// Starting identification.
	result, err := StreamingIdentify(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming identification: %v\n", err)
		os.Exit(1)
	}

	// Server returns the index of the voiceprint that matches the best, a similarity
	// score for each voiceprint along with whether the score exceeded the server-configured
	// threshold for being a match.
	//
	// If none of the voiceprints were a good match, the best match index will be negative.
	matched := "❌ No Match found"
	if result.BestMatchIndex >= 0 {
		bestScore := result.VoiceprintComparisonResults[result.BestMatchIndex].SimilarityScore
		matched = fmt.Sprintf("✅ Match found: Index: %d, Score: %1.3f", result.BestMatchIndex, bestScore)
	}

	fmt.Printf("\nIdentification Result:\n")

	fmt.Printf("Scores:\n")
	for i, r := range result.VoiceprintComparisonResults {
		fmt.Printf("Index: %d, Score: %1.3f, IsMatch: %v\n", i, r.SimilarityScore, r.IsMatch)
	}

	fmt.Printf("\n%s\n", matched)
}

// StreamingIdentify wraps the streaming API for performing speaker identification
// using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingIdentify(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.IdentificationConfig,
	audio io.Reader,
) (*voicebio.StreamingIdentifyResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingIdentify(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingIdentifyClient,
	cfg *voicebio.IdentificationConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingIdentifyRequest{
		Request: &voicebio.StreamingIdentifyRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingIdentifyRequest{
				Request: &voicebio.StreamingIdentifyRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

Streaming from microphone

  • Streaming audio from microphone input basically requires a reader interface that can provided audio samples recorded from a microphone; typically this requires interaction with system libraries. Another option is to use an external command line tool like sox to record and pipe audio into the client.

  • The examples below use the latter approach by using the rec command provided with sox to record and stream the audio.

#!/usr/bin/env python3

# This example assumes sox is installed on the system and is available
# in the system's PATH variable. Instead of opening a regular file from
# disk, we open a subprocess that executes sox's rec command to record
# audio from the system's default microphone.

import subprocess
import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example.
m = modelResp.models[0]
modelID = m.id

# Loading reference voiceprints.
voiceprints = []
for p in ["user1.bin", "user2.bin", "user3.bin"]:
    with open(p, 'r') as f:
        voiceprints.append(voicebio.Voiceprint(data=f.read().strip()))

# Setting audio format to be raw 16-bit signed little endian audio samples
# recorded at the sample rate expected by the model.
cfg = voicebio.IdentificationConfig(
    model_id=modelID,
    voiceprints=voiceprints,
    audio_format=voicebio.AudioFormat(
      audio_format_raw=voicebio.AudioFormatRAW(
        encoding="AUDIO_ENCODING_SIGNED",
        bit_depth=16,
        byte_order="BYTE_ORDER_LITTLE_ENDIAN",
        sample_rate=m.attributes.sample_rate,
        channels=1,
      )
    ),
)

# Open microphone stream using sox's rec command and record
# audio using the config specified above for *10 seconds*.
maxDuration = 10
cmd = f"rec -t raw -r {m.attributes.sample_rate} -e signed -b 16 -L -c 1 - trim 0 {maxDuration}"
mic = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
audio = mic.stdout

try:
  _ = audio.read(1024) # Trying to read some bytes as sanity check.
except Exception as err:
    print(f"[ERROR] failed to read audio from mic stream: {err}")

print(f"\n[INFO] recording {maxDuration} seconds of audio microphone ... \n")

# The first request to the server should only contain the
# recognition configuration. Subsequent requests should contain
# audio bytes. We can write a simple generator to do this.
def stream(cfg, audio, bufferSize=1024):
    yield voicebio.StreamingIdentifyRequest(config=cfg)

    data = audio.read(bufferSize)
    while len(data) > 0:
        yield voicebio.StreamingIdentifyRequest(audio=voicebio.Audio(data=data))
        data = audio.read(bufferSize)

# Streaming audio to the server.
result = client.StreamingIdentify(stream(cfg, audio))

# Server returns the index of the voiceprint that matches the best, a similarity
# score for each voiceprint along with whether the score exceeded the server-configured
# threshold for being a match.
#
# If none of the voiceprints were a good match, the best match index will be negative.
matched = "❌ No Match found"
if result.best_match_index >= 0:
    best_score = result.voiceprint_comparison_results[result.best_match_index].similarity_score
    matched = f"✅ Match found: Index: {result.best_match_index}, Score: {best_score:1.3f}"

print(f"\nIdentification Result:\n")

print("Scores:")
for i, r in enumerate(result.voiceprint_comparison_results):
    print(f"Index: {i}, Score: {r.similarity_score:1.3f}, IsMatch: {r.is_match}")

print(f"\n{matched}")

audio.close()
mic.kill()
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"strings"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Selecting first model.
	m := modelResp.Models[0]

	// Reading voiceprint data.
	voiceprints := make([]*voicebio.Voiceprint, 0)

	for i, p := range []string{"user1.bin", "user2.bin", "user3.bin"} {
		data, err := os.ReadFile(p)
		if err != nil {
			fmt.Printf("\nfailed to read voiceprint[%d] data: %v\n", i, err)
			os.Exit(1)
		}

		voiceprints = append(voiceprints, &voicebio.Voiceprint{Data: string(data)})
	}

	// Setting audio format to be raw 16-bit signed little endian audio samples
	// recorded at the sample rate expected by the model.
	cfg := &voicebio.IdentificationConfig{
		ModelId:     m.Id,
		Voiceprints: voiceprints,
		AudioFormat: &voicebio.AudioFormat{AudioFormat: &voicebio.AudioFormat_AudioFormatRaw{
			AudioFormatRaw: &voicebio.AudioFormatRAW{
				Encoding:   voicebio.AudioEncoding_AUDIO_ENCODING_SIGNED,
				SampleRate: m.Attributes.SampleRate,
				BitDepth:   16,
				ByteOrder:  voicebio.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
				Channels:   1,
			},
		},
		},
	}

	// Open microphone stream using sox's rec command and record
	// audio using the config specified above for *10 seconds*.
	maxDuration := 10
	args := fmt.Sprintf("-t raw -r %d -e signed -b 16 -L -c 1 - trim 0 %d", m.Attributes.SampleRate, maxDuration)
	cmd := exec.CommandContext(ctx, "rec", strings.Fields(args)...)
	cmd.Stderr = os.Stderr

	audio, err := cmd.StdoutPipe()
	if err != nil {
		fmt.Printf("failed to open microphone stream: %v\n", err)
		os.Exit(1)
	}

	// Starting routines to record from microphone and stream to server
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(ctx)

	eg.Go(func() error {
		fmt.Printf("\n[INFO] recording %d seconds from microphone \n", maxDuration)

		if err := cmd.Run(); err != nil {
			return fmt.Errorf("record from microphone: %w", err)
		}

		return nil
	})

	// Starting identification.
	result, err := StreamingIdentify(ctx, client, cfg, audio)
	if err != nil {
		fmt.Printf("failed to run streaming identification: %v\n", err)
		os.Exit(1)
	}

	// Server returns the index of the voiceprint that matches the best, a similarity
	// score for each voiceprint along with whether the score exceeded the server-configured
	// threshold for being a match.
	//
	// If none of the voiceprints were a good match, the best match index will be negative.
	matched := "❌ No Match found"
	if result.BestMatchIndex >= 0 {
		bestScore := result.VoiceprintComparisonResults[result.BestMatchIndex].SimilarityScore
		matched = fmt.Sprintf("✅ Match found: Index: %d, Score: %1.3f", result.BestMatchIndex, bestScore)
	}

	fmt.Printf("\nIdentification Result:\n")

	fmt.Printf("Scores:\n")
	for i, r := range result.VoiceprintComparisonResults {
		fmt.Printf("Index: %d, Score: %1.3f, IsMatch: %v\n", i, r.SimilarityScore, r.IsMatch)
	}

	fmt.Printf("\n%s\n", matched)
}

// StreamingIdentify wraps the streaming API for performing speaker identification
// using the given cfg.
//
// Data is read from the given audio reader into a buffer and streamed to VoiceBio
// server. The default buffer size may be overridden using Options when creating
// the Client.
//
// If any error occurs while reading the audio or sending it to the server, this
// method will immediately exit, returning that error.
func StreamingIdentify(
	ctx context.Context,
	client voicebio.VoiceBioServiceClient,
	cfg *voicebio.IdentificationConfig,
	audio io.Reader,
) (*voicebio.StreamingIdentifyResponse, error) {
	const (
		streamingBufSize = 1024
	)

	// Creating stream.
	stream, err := client.StreamingIdentify(ctx)
	if err != nil {
		return nil, err
	}

	// Sending audio.
	if err := sendAudio(stream, cfg, audio, streamingBufSize); err != nil && !errors.Is(err, io.EOF) {
		// if sendAudio encountered io.EOF, it's only a
		// notification that the stream has closed.  The actual
		// status will be obtained in the CloseAndRecv call. We
		// therefore return on non-EOF errors here.
		return nil, err
	}

	// Returning result.
	return stream.CloseAndRecv()
}

// sendAudio sends the config and audio to a stream.
func sendAudio(
	stream voicebio.VoiceBioService_StreamingIdentifyClient,
	cfg *voicebio.IdentificationConfig,
	audio io.Reader,
	bufSize uint32,
) error {
	// The first message needs to be a config message, and all subsequent
	// messages must be audio messages.

	// Send the config.
	if err := stream.Send(&voicebio.StreamingIdentifyRequest{
		Request: &voicebio.StreamingIdentifyRequest_Config{Config: cfg},
	}); err != nil {
		// if this failed, we don't need to CloseSend
		return err
	}

	// Stream the audio.
	buf := make([]byte, bufSize)
	for {
		n, err := audio.Read(buf)
		if n > 0 {
			if err2 := stream.Send(&voicebio.StreamingIdentifyRequest{
				Request: &voicebio.StreamingIdentifyRequest_Audio{
					Audio: &voicebio.Audio{Data: buf[:n]},
				},
			}); err2 != nil {
				// if we couldn't Send, the stream has
				// encountered an error and we don't need to
				// CloseSend.
				return err2
			}
		}

		if err != nil {
			// err could be io.EOF, or some other error reading from
			// audio.  In any case, we need to CloseSend, send the
			// appropriate error to errCh and return from the function
			if err2 := stream.CloseSend(); err2 != nil {
				return err2
			}

			if err != io.EOF {
				return err
			}

			return nil
		}
	}
}

2.1.7 - Comparing Voiceprints

Describes how to compare pre-extracted voiceprints using VoiceBio’s CompareVoiceprints API.
  • The CompareVoiceprints endpoint allows the user to compare pre-extracted voiceprints and get similarity scores and match results without needing to send audio data.

  • This is useful in cases where the user wants to compare a given voiceprint against a large number of other voiceprints, and sending audio data for each comparison would be inefficient. The client can enroll the voiceprint once using the StreamingEnroll method, and then use this method to compare it against a large number of other voiceprints in batches.

  • The following example shows how to compare pre-extracted voiceprints using VoiceBio’s CompareVoiceprints API, without streaming audio. The voiceprints can be loaded from files on disk or obtained from previous enrollment sessions.

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example. The model ID should be the same as the one used to
# generate the voiceprints being compared.
modelID = modelResp.models[0].id

# Loading reference voiceprints.
reference_voiceprints = []
for p in ["user1.bin", "user2.bin", "user3.bin"]:
    with open(p, 'r') as f:
        reference_voiceprints.append(voicebio.Voiceprint(data=f.read().strip()))

# Load the target voiceprint that we want to compare against the reference voiceprints.
with open("unknown.bin", 'r') as f:
    target_voiceprint = voicebio.Voiceprint(data=f.read().strip())

# Set the comparison config.
req = voicebio.CompareVoiceprintsRequest(
    model_id=modelID,
    target_voiceprint=target_voiceprint,
    reference_voiceprints=reference_voiceprints,
)

# Compare voiceprints.
result = client.CompareVoiceprints(req)

# Server returns the index of the voiceprint that matches the best, a similarity
# score for each voiceprint along with whether the score exceeded the server-configured
# threshold for being a match.
#
# If none of the voiceprints were a good match, the best match index will be negative.
matched = "❌ No Match found"
if result.best_match_index >= 0:
    best_score = result.voiceprint_comparison_results[result.best_match_index].similarity_score
    matched = f"✅ Match found: Index: {result.best_match_index}, Score: {best_score:1.3f}"

print(f"\nComparison Result:\n")

print("Scores:")
for i, r in enumerate(result.voiceprint_comparison_results):
    print(f"Index: {i}, Score: {r.similarity_score:1.3f}, IsMatch: {r.is_match}")

print(f"\n{matched}")
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Reading voiceprint data.
	reference_voiceprints := make([]*voicebio.Voiceprint, 0)

	for i, p := range []string{"user1.bin", "user2.bin", "user3.bin"} {
		data, err := os.ReadFile(p)
		if err != nil {
			fmt.Printf("\nfailed to read voiceprint[%d] data: %v\n", i, err)
			os.Exit(1)
		}

		reference_voiceprints = append(reference_voiceprints, &voicebio.Voiceprint{Data: string(data)})
	}

	// Load the target voiceprint that we want to compare against the reference voiceprints.
	data, err := os.ReadFile("unknown.bin")
	if err != nil {
		fmt.Printf("failed to read target voiceprint data: %v\n", err)
		os.Exit(1)
	}

	target_voiceprint := &voicebio.Voiceprint{Data: string(data)}

	// Selecting the first model. The model ID should be the same as the one used to generate the
	// voiceprints being compared.
	req := &voicebio.CompareVoiceprintsRequest{
		ModelId:            modelResp.Models[0].Id,
		TargetVoiceprint:   target_voiceprint,
		ReferenceVoiceprints: reference_voiceprints,
	}

	// Compare voiceprints.
	result, err := client.CompareVoiceprints(ctx, req)
	if err != nil {
		fmt.Printf("failed to compare voiceprints: %v\n", err)
		os.Exit(1)
	}

	// Server returns the index of the voiceprint that matches the best, a similarity
	// score for each voiceprint along with whether the score exceeded the server-configured
	// threshold for being a match.
	//
	// If none of the voiceprints were a good match, the best match index will be negative.
	matched := "❌ No Match found"
	if result.BestMatchIndex >= 0 {
		bestScore := result.VoiceprintComparisonResults[result.BestMatchIndex].SimilarityScore
		matched = fmt.Sprintf("✅ Match found: Index: %d, Score: %1.3f", result.BestMatchIndex, bestScore)
	}

	fmt.Printf("\n Comparison Result:\n")

	fmt.Printf("Scores:\n")
	for i, r := range result.VoiceprintComparisonResults {
		fmt.Printf("Index: %d, Score: %1.3f, IsMatch: %v\n", i, r.SimilarityScore, r.IsMatch)
	}

	fmt.Printf("\n%s\n", matched)
}

2.1.8 - Vectorizing Voiceprints

Describes how to convert voiceprints into vector representations using VoiceBio’s VectorizeVoiceprints API.
  • Voiceprints can also be vectorized using the VectorizeVoiceprints API, which returns a vector representation of each voiceprint that can be used for downstream tasks such as clustering, custom scoring, other machine learning models or even semantic searching in vector databases.

  • See the API reference for more details.

  • The following example shows how to use the VectorizeVoiceprints API to vectorize voiceprints. The voiceprints can be loaded from files on disk or obtained from previous enrollment sessions.

import numpy as np

import grpc
import cobaltspeech.voicebio.v1.voicebio_pb2_grpc as stub
import cobaltspeech.voicebio.v1.voicebio_pb2 as voicebio

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceBioServiceStub(channel)

# Get server version.
versionResp = client.Version(voicebio.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicebio.ListModelsRequest())

print("Models:")
for model in modelResp.models:
    print(model)

# Select a model ID from the list above. Going with the first model
# in this example. The model ID should be the same as the one used to
# generate the voiceprints being vectorized.
modelID = modelResp.models[0].id

# Loading voiceprints.
voiceprints = []
for p in ["user1.bin", "user2.bin", "user3.bin"]:
    with open(p, 'r') as f:
        voiceprints.append(voicebio.Voiceprint(data=f.read().strip()))

# Set the vectorization config.
req = voicebio.VectorizeVoiceprintsRequest(
    model_id=modelID,
    voiceprints=voiceprints,
)

# Vectorize voiceprints.
result = client.VectorizeVoiceprints(req)

# The server returns a list of vectorized voiceprints in the same order as the input voiceprints.
#
# In most cases, the vectorized voiceprints can be compared using simple distance metrics such as
# cosine similarity or euclidean distance. This is not guaranteed, however, and depends on the model
# used to generate the voiceprints and vectorize them.

# Example using cosine similarity.
n = len(result.voiceprints)
similarity = np.zeros((n, n), dtype=np.float32)

for i, vi in enumerate(result.voiceprints):
  for j, vj in enumerate(result.voiceprints):
    similarity[i, j] = np.dot(vi.data, vj.data) / (np.linalg.norm(vi.data) * np.linalg.norm(vj.data))

print("Cosine Similarity Matrix:")
print(similarity)
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicebio "github.com/cobaltspeech/go-genproto/cobaltspeech/voicebio/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicebio.NewVoiceBioServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicebio.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicebio.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Reading voiceprint data.
	voiceprints := make([]*voicebio.Voiceprint, 0)

	for i, p := range []string{"user1.bin", "user2.bin", "user3.bin"} {
		data, err := os.ReadFile(p)
		if err != nil {
			fmt.Printf("\nfailed to read voiceprint[%d] data: %v\n", i, err)
			os.Exit(1)
		}

		voiceprints = append(voiceprints, &voicebio.Voiceprint{Data: string(data)})
	}

	// Selecting the first model. The model ID should be the same as the one used to generate the
	// voiceprints being compared.
	req := &voicebio.VectorizeVoiceprintsRequest{
		ModelId:            modelResp.Models[0].Id,
		Voiceprints:        voiceprints,
	}

	// Vectorize voiceprints.
	result, err := client.VectorizeVoiceprints(ctx, req)
	if err != nil {
		fmt.Printf("failed to vectorize voiceprints: %v\n", err)
		os.Exit(1)
	}

	// The server returns a list of vectorized voiceprints in the same order as the input voiceprints.
	//
	// In almost cases, the vectorized voiceprints can be compared using simple distance metrics such as
	// cosine similarity or euclidean distance. This  is not guaranteed, however, and depends on the model
	// used to generate the voiceprints and vectorize them.

	// Example using cosine similarity.
	n := len(result.Voiceprints)
	similarity := make([][]float32, n)
	for i := range similarity {
		similarity[i] = make([]float32, n)
	}

	for i, vi := range result.Voiceprints {
		for j, vj := range result.Voiceprints {
			dotProduct := float32(0.0)
			normVi := float32(0.0)
			normVj := float32(0.0)

			for k := range vi.Data {
				dotProduct += vi.Data[k] * vj.Data[k]
				normVi += vi.Data[k] * vi.Data[k]
				normVj += vj.Data[k] * vj.Data[k]
			}

            denom := float32(math.Sqrt(float64(normVi)) * math.Sqrt(float64(normVj)))
			similarity[i][j] = dotProduct / denom
		}
	}

	fmt.Printf("Cosine Similarity Matrix:\n")
	for i := range similarity {
		for j := range similarity[i] {
			fmt.Printf("%1.3f ", similarity[i][j])
		}

		fmt.Println()
    }
}

2.1.9 - API Reference

Detailed reference for API requests and types.

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

VoiceBioService

Service that implements the Cobalt VoiceBio API.

Version

Version(VersionRequest) VersionResponse

Returns version information from the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

Returns information about the models available on the server.

StreamingEnroll

StreamingEnroll(StreamingEnrollRequest) StreamingEnrollResponse

Uses new audio data to perform enrollment of new users, or to update enrollment of existing users. Returns a new or updated voiceprint.

Clients should store the returned voiceprint against the ID of the user that provided the audio. This voiceprint can be provided later, with the Verify or Identify requests to match new audio against known speakers.

If this call is used to update an existing user’s voiceprint, the old voiceprint can be discarded and only the new one can be stored for that user.

StreamingVerify

StreamingVerify(StreamingVerifyRequest) StreamingVerifyResponse

Compares audio data against the provided voiceprint and verifies whether or not the audio matches against the voiceprint.

StreamingIdentify

StreamingIdentify(StreamingIdentifyRequest) StreamingIdentifyResponse

Compares audio data against the provided list of voiceprints and identifies which (or none) of the voiceprints is a match for the given audio.

If you have a large list of voiceprints that you want to compare against in batches, and you want to avoid sending audio data for each comparison, you can use the CompareVoiceprints method instead, which operates on pre-extracted voiceprints without requiring any audio data. You would first need to extract the voiceprint from the audio using the StreamingEnroll method, and then use that voiceprint in the CompareVoiceprints method as the target voiceprint.

VectorizeVoiceprints

VectorizeVoiceprints(VectorizeVoiceprintsRequest) VectorizeVoiceprintsResponse

Converts the given voiceprints into numerical vector representations that can be used for various downstream tasks such as clustering, visualization, or as input features for other machine learning models. The specific format and dimensionality of these vectors may vary depending on the model used.

CompareVoiceprints

CompareVoiceprints(CompareVoiceprintsRequest) CompareVoiceprintsResponse

Compares pre-extracted voiceprints and returns similarity scores and match results without needing to send audio data. This is useful in cases where the user wants to compare a given voiceprint against a large number of other voiceprints, and sending audio data for each comparison would be inefficient. The client can enroll the voiceprint once using the StreamingEnroll method, and then use this method to compare it against a large number of other voiceprints in batches.

Messages

  • If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
  • If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

Audio

Audio to be sent to VoiceBio.

Fields

AudioFormat

Format of the audio to be sent for recognition.

Depending on how they are configured, server instances of this service may not support all the formats provided in the API. One format that is guaranteed to be supported is the RAW format with little-endian 16-bit signed samples with the sample rate matching that of the model being requested.

Fields

  • oneof audio_format.audio_format_raw (AudioFormatRAW ) Audio is raw data without any headers

  • oneof audio_format.audio_format_headered (AudioFormatHeadered ) Audio has a self-describing header. Headers are expected to be sent at the beginning of the entire audio file/stream, and not in every Audio message.

    The default value of this type is AUDIO_FORMAT_HEADERED_UNSPECIFIED. If this value is used, the server may attempt to detect the format of the audio. However, it is recommended that the exact format be specified.

AudioFormatRAW

Details of audio in raw format

Fields

  • encoding (AudioEncoding ) Encoding of the samples. It must be specified explicitly and using the default value of AUDIO_ENCODING_UNSPECIFIED will result in an error.

  • bit_depth (uint32 ) Bit depth of each sample (e.g. 8, 16, 24, 32, etc.). This is a required field.

  • byte_order (ByteOrder ) Byte order of the samples. This field must be set to a value other than BYTE_ORDER_UNSPECIFIED when the bit_depth is greater than 8.

  • sample_rate (uint32 ) Sampling rate in Hz. This is a required field.

  • channels (uint32 ) Number of channels present in the audio. E.g.: 1 (mono), 2 (stereo), etc. This is a required field.

CompareVoiceprintsRequest

The top level message sent by the client for the CompareVoiceprints method. This is similar to StreamingIdentifyRequest, but operates on pre-extracted voiceprints without sending any audio data.

Fields

  • model_id (string ) ID of the model to use for comparison. The model used for comparison must match with the model used for enrollment of the voiceprints. A list of supported IDs can be found using the ListModels call.

  • target_voiceprint (Voiceprint ) The voiceprint to compare against the reference voiceprints.

  • reference_voiceprints (Voiceprint repeated) Voiceprints that should be compared against the target voiceprint.

CompareVoiceprintsResponse

The message returned by the server for the CompareVoiceprints method. This contains the similarity scores and match results for comparing the target voiceprint against each of the reference voiceprints, as well as the index of the best matching voiceprint in the reference list, if any of them is a match. This is similar to StreamingIdentifyResponse, but operates on pre-extracted voiceprints without sending any audio data.

Fields

  • best_match_index (int32 ) Index (0-based) of the best matching voiceprint in the list of reference voiceprints provided in the CompareVoiceprintsRequest message. If none of the voiceprints was a match, a negative value is returned.

  • voiceprint_comparison_results (VoiceprintComparisonResult repeated) Result of comparing the given the target voiceprint against each of the reference voiceprints. The order of this list is the same as the reference voiceprint list provided in the CompareVoiceprintsRequest message.

EnrollmentConfig

Configuration for Enrollment of speakers.

Fields

  • model_id (string ) ID of the model to use for enrollment. A list of supported IDs can be found using the ListModels call.

  • audio_format (AudioFormat ) Format of the audio to be sent for enrollment.

  • previous_voiceprint (Voiceprint ) Empty string for new users. For re-enrolling additional users with new audio data, set this to that user’s previous voiceprint. The previous voiceprint needs to have been generated using the same model as specified in this config.

EnrollmentStatus

The message returned as part of StreamingEnrollResponse, to provide information about whether voiceprint is sufficiently trained.

Fields

  • enrollment_complete (bool ) Whether sufficient data has been provided as part of this user’s enrollment. If this is false, more audio should be collected from the user and re-enrollment should be done. If this is true, it is still OK to enroll more data for the same user to update the voiceprint.

  • additional_audio_required_seconds (uint32 ) If enrollment is not yet complete, how many more seconds of user’s speech are required to complete the enrollment. If enrollment is completed successfully, this value will be set to 0.

IdentificationConfig

Configuration for Identification of a speaker.

Fields

  • model_id (string ) ID of the model to use for identification. A list of supported IDs can be found using the ListModels call. The model used for identification must match with the model used for enrollment.

  • audio_format (AudioFormat ) Format of the audio to be sent for enrollment.

  • voiceprints (Voiceprint repeated) Voiceprints of potential speakers that need to be identified in the given audio.

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

  • models (Model repeated) List of models available for use.

Model

Description of a VoiceBio model.

Fields

  • id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for enrollment, verification or identification requests. This ID needs to be specified in the respective config messages for these requests.

  • name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their voicebio task.

  • attributes (ModelAttributes ) Model Attributes

ModelAttributes

Attributes of a VoiceBio model

Fields

  • sample_rate (uint32 ) Audio sample rate (native) supported by the model

StreamingEnrollRequest

The top level messages sent by the client for the StreamingEnroll method. In this streaming call, multiple StreamingEnrollRequest messages should be sent. The first message must contain a EnrollmentConfig message, and all subsequent messages must contain Audio only. All Audio messages must contain non-empty audio. If audio content is empty, the server may choose to interpret it as end of stream and stop accepting any further messages.

Fields

StreamingEnrollResponse

The message returned by the server for the StreamingEnroll method.

Fields

StreamingIdentifyRequest

The top level messages sent by the client for the StreamingIdentify method. In this streaming call, multiple StreamingIdentifyRequest messages should be sent. The first message must contain a IdentificationConfig message, and all subsequent messages must contain Audio only. All Audio messages must contain non-empty audio. If audio content is empty, the server may choose to interpret it as end of stream and stop accepting any further messages.

Fields

StreamingIdentifyResponse

The message returned by the server for the StreamingIdentify method.

Fields

  • best_match_index (int32 ) Index (0-based) of the best matching voiceprint in the list of input voiceprints provided in the IdentificationConfig message. If none of the voiceprints was a match, a negative value is returned.

  • voiceprint_comparison_results (VoiceprintComparisonResult repeated) Result of comparing the given audio against each of the input voiceprints. The order of this list is the same as the input voiceprint list provided in the IdentificationConfig message.

StreamingVerifyRequest

The top level messages sent by the client for the StreamingVerify method. In this streaming call, multiple StreamingVerifyRequest messages should be sent. The first message must contain a VerificationConfig message, and all subsequent messages must contain Audio only. All Audio messages must contain non-empty audio. If audio content is empty, the server may choose to interpret it as end of stream and stop accepting any further messages.

Fields

StreamingVerifyResponse

The message returned by the server for the StreamingVerify method.

Fields

VectorVoiceprint

Voiceprint represented in vector form. The specific format and dimensionality of this vector may vary depending on the model used. The VectorizeVoiceprints method can be used to convert a Voiceprint into a VectorVoiceprint representation.

Fields

  • data (float repeated) List of floating point values representing the voiceprint in vector form.

VectorizeVoiceprintsRequest

The top level message sent by the client for the VectorizeVoiceprints method.

Fields

  • model_id (string ) ID of the model to use for vectorization. The model used for vectorization must match with the model used for enrollment of the voiceprints. A list of supported IDs can be found using the ListModels call.

  • voiceprints (Voiceprint repeated) Voiceprints to be vectorized.

VectorizeVoiceprintsResponse

The message returned by the server for the VectorizeVoiceprints method.

Fields

  • voiceprints (VectorVoiceprint repeated) Voiceprint data converted into a vector representation, which can be used for various downstream tasks such as clustering, visualization, or as input features for other machine learning models. The specific format and dimensionality of these vectors may vary depending on the model used.

    The order of this list is the same as the input voiceprint list provided in the VectorizeVoiceprintsRequest message.

VerificationConfig

Configuration for Verification of a speaker.

Fields

  • model_id (string ) ID of the model to use for verification. A list of supported IDs can be found using the ListModels call. The model used for verification must match with the model used for enrollment.

  • audio_format (AudioFormat ) Format of the audio to be sent for enrollment.

  • voiceprint (Voiceprint ) Voiceprint with which audio should be compared.

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The message sent by the server for the Version method.

Fields

  • version (string ) Version of the server handling these requests.

Voiceprint

Voiceprint extracted from user’s audio.

Fields

  • data (string ) Voiceprint data serialized to a string.

VoiceprintComparisonResult

Message describing the result of comparing a voiceprint against given audio.

Fields

  • is_match (bool ) Whether or not the audio successfully matches with the provided voiceprint.

  • similarity_score (float ) Similarity score representing how closely the audio matched against the voiceprint. This score could be any negative or positive number. Lower value suggests that the audio and voiceprints are less similar, whereas a higher value indicates more similarity. The is_match field can be used to actually decide if the result should be considered a valid match.

Enums

AudioEncoding

The encoding of the audio data to be sent for recognition.

NameNumberDescription
AUDIO_ENCODING_UNSPECIFIED0AUDIO_ENCODING_UNSPECIFIED is the default value of this type and will result in an error.
AUDIO_ENCODING_SIGNED1PCM signed-integer
AUDIO_ENCODING_UNSIGNED2PCM unsigned-integer
AUDIO_ENCODING_IEEE_FLOAT3PCM IEEE-Float
AUDIO_ENCODING_ULAW4G.711 mu-law
AUDIO_ENCODING_ALAW5G.711 a-law

AudioFormatHeadered

NameNumberDescription
AUDIO_FORMAT_HEADERED_UNSPECIFIED0AUDIO_FORMAT_HEADERED_UNSPECIFIED is the default value of this type.
AUDIO_FORMAT_HEADERED_WAV1WAV with RIFF headers
AUDIO_FORMAT_HEADERED_MP32MP3 format with a valid frame header at the beginning of data
AUDIO_FORMAT_HEADERED_FLAC3FLAC format
AUDIO_FORMAT_HEADERED_OGG_OPUS4Opus format with OGG header

ByteOrder

Byte order of multi-byte data

NameNumberDescription
BYTE_ORDER_UNSPECIFIED0BYTE_ORDER_UNSPECIFIED is the default value of this type.
BYTE_ORDER_LITTLE_ENDIAN1Little Endian byte order
BYTE_ORDER_BIG_ENDIAN2Big Endian byte order

Scalar Value Types

.proto TypeC++ TypeC# TypeGo TypeJava TypePHP TypePython TypeRuby Type

double
doubledoublefloat64doublefloatfloatFloat

float
floatfloatfloat32floatfloatfloatFloat

int32
int32intint32intintegerintBignum or Fixnum (as required)

int64
int64longint64longinteger/stringint/longBignum

uint32
uint32uintuint32intintegerint/longBignum or Fixnum (as required)

uint64
uint64ulonguint64longinteger/stringint/longBignum or Fixnum (as required)

sint32
int32intint32intintegerintBignum or Fixnum (as required)

sint64
int64longint64longinteger/stringint/longBignum

fixed32
uint32uintuint32intintegerintBignum or Fixnum (as required)

fixed64
uint64ulonguint64longinteger/stringint/longBignum

sfixed32
int32intint32intintegerintBignum or Fixnum (as required)

sfixed64
int64longint64longinteger/stringint/longBignum

bool
boolboolboolbooleanbooleanbooleanTrueClass/FalseClass

string
stringstringstringStringstringstr/unicodeString (UTF-8)

bytes
stringByteString[]byteByteStringstringstrString (ASCII-8BIT)

3 - Speech Assessment

3.1 - CAPT

On-prem / on-device pronunciation assessment: scores how accurately a speaker pronounced a known piece of text.

CAPT (Computer-Aided Pronunciation Training) answers a different question from a speech recognizer. Transcribe is given audio and asked what was said. CAPT is given audio and the text the speaker was supposed to say, and asked how closely the audio matches it, phoneme by phoneme.

That difference matters whenever you already know the target. Reading assessment, pronunciation practice, language learning drills and scripted-prompt verification all share the same shape: a known reference, a spoken attempt, and a decision about whether the attempt was good enough.

Because the reference is known, CAPT reports where the attempt diverged, not just that it did:

  • Every phoneme of the reference is aligned to the audio and labeled MATCH, SUBSTITUTION, DELETION or INSERTION.
  • Every aligned phoneme carries a score in [0, 1], derived from the acoustic model’s confidence and remapped through per-phoneme thresholds you control.
  • Every recognized phoneme carries a start time and duration, so results can be laid over a waveform.
  • Words are scored as groups of phonemes, and the utterance gets a single overall score.

Crucially, CAPT is error-preserving. A general-purpose recognizer (and a large end-to-end model especially) is built to recover the intended message, so it will quietly repair a mispronunciation into the word it assumes you meant. That behavior is exactly wrong for assessment. CAPT never substitutes what it expects for what it heard: if the speaker said something other than the reference, the divergence is reported.

The engine runs on your own hardware: on-premise, in your private cloud, or embedded on a device. Audio never leaves your infrastructure.

How it works

CAPT is built on top of Cobalt Transcribe, running a phoneme-level acoustic model:

  1. The reference text is phonemized. Each word is looked up in a pronunciation lexicon; words that are not in it are passed to a grapheme-to-phoneme (G2P) model. A word may legitimately have several valid pronunciations, and all of them are considered.
  2. The audio is recognized into a confusion network: for each slice of time, a probability distribution over the phonemes that might have been spoken, rather than a single guess.
  3. Reference and audio are aligned with a weighted edit distance, producing the per-phoneme MATCH / SUBSTITUTION / DELETION / INSERTION labels.
  4. Each phoneme is scored from its confidence in the confusion network, remapped through the cutoffs you configure.

Keeping the full distribution rather than a single best guess is what makes step 4 meaningful: the score for a phoneme reflects how much probability mass the acoustic model actually put on it.

Where to start

3.1.1 - Getting Started

How to get a CAPT server running on your system

Using Cobalt CAPT

  • A typical CAPT release, provided as a compressed archive, contains a linux binary (capt-server) for the required native CPU architecture, an appropriate Dockerfile, and models.

  • Cobalt CAPT runs either locally on linux or using Docker.

  • Cobalt CAPT serves the CAPT gRPC API on port 2727. The configuration shipped with a release also enables an HTTP+JSON gateway on port 8080 and operational endpoints on port 8081; both are opt-in settings rather than server defaults, so a config written from scratch has gRPC only.

  • To quickly try out CAPT, first start the server as shown below and use the SDK in your preferred language to call it from your application.

Running CAPT Server Locally on Linux

./capt-server --config capt-server.cfg.toml

By default the binary assumes a configuration file named capt-server.cfg.toml in the same directory. A different config file may be specified using the --config argument.

On a successful start the server logs the addresses it is serving on:

2026/09/10 03:57:50 info  {"msg":"server initializing"}
2026/09/10 03:57:50 info  {"msg":"license verified"}
2026/09/10 03:57:51 info  {"source":"transcribe","msg":"formats supported","formats":"[RAW WAV FLAC MP3 Opus]"}
2026/09/10 03:57:51 info  {"msg":"runtime initialized","model_count":"1","init_time_taken":"1.485372717s"}
2026/09/10 03:57:51 info  {"msg":"server started","grpcAddr":"[::]:2727","httpApiAddr":"[::]:8080","httpOpsAddr":"[::]:8081"}

Running CAPT Server as a Docker Container

To build and run the Docker image for CAPT, run:

docker build -t cobalt-capt .
docker run -p 2727:2727 -p 8080:8080 cobalt-capt

Checking the server is up

The HTTP API is the quickest way to confirm a working server, provided api.Address is set in your config as the shipped one does. It exposes the unary calls: Version, ListModels and Evaluate. The bidirectional StreamingEvaluate is gRPC-only, though browsers can reach it over a websocket:

curl -s http://localhost:8080/api/capt/v1/version
{"version":"1.4.0"}

The reported version is that of the capt-server release you were given.

curl -s http://localhost:8080/api/capt/v1/list-models
{
  "models": [
    {
      "id": "en_US-16khz",
      "name": "en_US CAPT (16khz audio)",
      "kind": "MODEL_KIND_SPEECH_EVALUATION",
      "attributes": {
        "sample_rate": 16000,
        "metadata": { "version": "0.0.0", "build_date": "unknown" }
      }
    }
  ]
}

The id of a model in this list is what you pass as model_id when configuring an evaluation, and attributes.sample_rate is the audio sample rate the model expects. See Evaluation Configurations for what to do with them.

How to Get a Copy of the CAPT Server and Models

Contact us for a release best suited to your requirements.

The release you receive is a compressed archive (tar.bz2), generally structured as follows:

release.tar.bz2
├── COPYING
├── README.md
├── capt-server
├── capt-server.cfg.toml
├── Dockerfile
├── api
│   ├── proto             [ cobaltspeech/capt/v1/capt.proto ]
│   └── gen               [ pre-generated Go and Python bindings ]
├── models
│   └── en_US-16khz
│       ├── capt          [ lexicon, G2P model, cutoffs, model config ]
│       └── transcribe    [ acoustic model and decoding graph ]
│
└── cobalt.license.key [ provided separately, needs to be copied over ]
  • The README.md file contains information about the release and instructions for starting the server on your system.

  • The capt-server is the server program, configured using the capt-server.cfg.toml file.

  • The Dockerfile can be used to create a container that will let you run CAPT server on non-linux systems such as macOS and Windows.

  • The api directory holds the protobuf definition of the API and pre-generated client bindings. CAPT’s proto is distributed with the release rather than from a public repository, so this is where it comes from. See Generating SDKs.

  • The models directory contains the evaluation models. Each model directory holds two halves: the transcribe acoustic model that recognizes phonemes, and the capt resources (pronunciation lexicon, G2P model and scoring cutoffs) that the evaluation is built from. Both are described in Tuning and Customization.

System Requirements

Cobalt CAPT runs on Linux, directly as a native application. You can evaluate the product on Windows or macOS using Docker Desktop, but we would not recommend that setup for production.

CAPT runs on x86_64 and Arm64 / aarch64 CPUs, and a statically linked build is available for deployment onto minimal or embedded images. Because the engine is CPU-only and the models are small, the same release can be deployed on a server, in a private cloud, or embedded on a device. Tell us the target and we will provide a build for it.

Sizing depends on the model and on how many evaluations you need to run concurrently. Please contact us for sizing guidance against your target hardware and expected load.

To integrate Cobalt CAPT into your application, follow the next steps to install or generate the SDK in a language of your choice.

3.1.2 - Generating SDKs

How to generate a client SDK for your project from the CAPT proto API definition.
  • The CAPT API is defined as a protocol buffer specification, a proto file. This file allows a developer to auto-generate client SDKs for a number of different programming languages.

  • The CAPT proto definition, cobaltspeech/capt/v1/capt.proto, is supplied to you as part of your release, together with pre-generated Go and Python bindings. If you only need those two languages you can skip straight to Using the pre-generated SDKs; to generate bindings for another language, see Generating SDKs.

The relevant part of your release looks like this:

release.tar.bz2
└── api
    ├── proto
    │   └── cobaltspeech
    │       └── capt
    │           └── v1
    │               └── capt.proto
    └── gen
        ├── go
        │   └── cobaltspeech/capt/v1/{capt.pb.go, capt_grpc.pb.go}
        └── py
            └── cobaltspeech/capt/v1/{capt_pb2.py, capt_pb2_grpc.py, capt_pb2.pyi}

Using the pre-generated SDKs

Golang

Copy the api/gen/go tree into your module, or add it as a dependency, and import the package:

import captpb "github.com/your-org/your-module/gen/go/cobaltspeech/capt/v1"

You will also need the gRPC and protobuf runtime libraries:

go get google.golang.org/protobuf
go get google.golang.org/grpc

Python

The Python bindings depend on Python >= 3.8. Place api/gen/py on your PYTHONPATH, or copy the cobaltspeech package into your project, and install the runtime dependencies:

pip install --upgrade pip
pip install --upgrade protobuf grpcio

Then import the modules:

import cobaltspeech.capt.v1.capt_pb2 as capt
import cobaltspeech.capt.v1.capt_pb2_grpc as capt_grpc

Generating SDKs from the proto

To generate bindings for a language we do not ship, generate them yourself from capt.proto. We recommend using buf, a command line tool that can generate documentation, schemas and SDK code for many languages.

Step 1. Installing buf

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/bin"

VERSION="1.72.0"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and add to $PATH.
chmod +x "${COBALT}/bin/buf"
export PATH="${PATH}:${COBALT}/bin"
brew install bufbuild/buf/buf

Step 2. Writing a buf.gen.yaml

Create a buf.gen.yaml next to the api/proto directory from your release. The example below generates Go and Python; other plugins can be added for more languages.

version: v1

managed:
  enabled: true
  go_package_prefix:
    default: github.com/your-org/your-module/gen

plugins:
  # Golang
  - plugin: buf.build/grpc/go
    out: gen/go
    opt: paths=source_relative

  - plugin: buf.build/protocolbuffers/go
    out: gen/go
    opt: paths=source_relative

  # Python
  - plugin: buf.build/grpc/python
    out: gen/py

  - plugin: buf.build/protocolbuffers/python
    out: gen/py

  - plugin: buf.build/protocolbuffers/pyi
    out: gen/py

Step 3. Generating code

# Removing any previously generated files.
rm -rf ./gen

# Generating code for the proto files inside the `proto` directory.
buf generate proto

You should now have a gen folder containing the generated code. The latest version of the CAPT API is v1. Import or copy the generated files into your project as per the conventions of your language.

gen
└── py
  └── cobaltspeech
    └── capt
      └── v1
        ├── capt_pb2_grpc.py
        ├── capt_pb2.py
        └── capt_pb2.pyi
gen
└── go
  └── cobaltspeech
    └── capt
      └── v1
        ├── capt_grpc.pb.go
        └── capt.pb.go

Once you have an SDK, continue to connecting to the server.

3.1.3 - Connecting to the Server

Describes how to connect to a running Cobalt CAPT server instance.

Once you have your CAPT server up and running, and have installed or generated the SDK for your project, you can connect to it by “dialing” a gRPC connection.

First, you need the address where the server is running: e.g. host:grpc_port. By default this is localhost:2727, and it is logged to the terminal when you start CAPT server as grpcAddr:

2026/09/10 03:57:51 info  {"msg":"server started","grpcAddr":"[::]:2727","httpApiAddr":"[::]:8080","httpOpsAddr":"[::]:8081"}

Default Connection

The following snippet connects to the server and queries its version, using an “insecure” gRPC channel. This would be the case if you have just started a local instance of CAPT server without TLS enabled.

import grpc
import cobaltspeech.capt.v1.capt_pb2 as capt
import cobaltspeech.capt.v1.capt_pb2_grpc as capt_grpc

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = capt_grpc.CAPTServiceStub(channel)

# Get server version.
versionResp = client.Version(capt.VersionRequest())
print(versionResp)

# Get the list of models available on the server.
modelResp = client.ListModels(capt.ListModelsRequest())
for model in modelResp.models:
    print(model)
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	captpb "github.com/your-org/your-module/gen/go/cobaltspeech/capt/v1"
)

func main() {
	const serverAddress = "localhost:2727"

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Using a channel without TLS enabled.
	conn, err := grpc.NewClient(serverAddress,
		grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}
	defer conn.Close()

	client := captpb.NewCAPTServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &captpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get the list of models available on the server.
	modelResp, err := client.ListModels(ctx, &captpb.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to list models: %v\n", err)
		os.Exit(1)
	}

	for _, m := range modelResp.GetModels() {
		fmt.Printf("%v\n", m)
	}
}

Connect with TLS

In our recommended setup for deployment, TLS is enabled in the gRPC connection, and clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

TLS is enabled server-side by providing a certificate and key in capt-server.cfg.toml:

[server.grpc]
Address = ":2727"
CertFile = "capt-server.crt"
KeyFile = "capt-server.key"

The following snippets show how to connect to a CAPT server that has TLS enabled.

import grpc
import cobaltspeech.capt.v1.capt_pb2 as capt
import cobaltspeech.capt.v1.capt_pb2_grpc as capt_grpc

serverAddress = "capt.your-org.internal:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = capt_grpc.CAPTServiceStub(channel)

# Get server version.
versionResp = client.Version(capt.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	captpb "github.com/your-org/your-module/gen/go/cobaltspeech/capt/v1"
)

func main() {
	const serverAddress = "capt.your-org.internal:2727"

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	conn, err := grpc.NewClient(serverAddress, grpc.WithTransportCredentials(creds))
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}
	defer conn.Close()

	client := captpb.NewCAPTServiceClient(conn)

	versionResp, err := client.Version(ctx, &captpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

In some setups it is desirable for the server to validate the clients connecting to it, and only respond to ones it can verify. If your CAPT server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

Note that in client-authentication mode the client still also verifies the server’s certificate, so this setup uses mutually authenticated TLS.

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)
// Root PEM certificate for validating a self-signed server certificate.
var rootCert []byte

// Client PEM certificate and private key.
var certPem, keyPem []byte

caCertPool := x509.NewCertPool()
if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
	fmt.Printf("unable to use given caCert\n")
	os.Exit(1)
}

clientCert, err := tls.X509KeyPair(certPem, keyPem)
if err != nil {
	fmt.Printf("unable to use given client certificate and key: %v\n", err)
	os.Exit(1)
}

tlsCfg := tls.Config{
	RootCAs:      caCertPool,
	Certificates: []tls.Certificate{clientCert},
}

creds := credentials.NewTLS(&tlsCfg)

3.1.4 - Evaluation Configurations

The options available when configuring an evaluation request.

Every evaluation begins with an EvaluationConfig. It is the first message of a StreamingEvaluate stream, and the config field of a unary Evaluate request. This page describes what goes in it.

Choosing a model

model_id selects the model to evaluate against, and must be one of the id values returned by ListModels. Each model also reports the sample_rate it expects and its kind:

  • MODEL_KIND_SPEECH_EVALUATION evaluates words and sentences. The reference_text is ordinary text.
  • MODEL_KIND_PHONEME_EVALUATION evaluates phonemes produced in isolation, which a word-oriented model may not recognize correctly. The reference_text is a sequence of phones. See Phoneme Models.

Both kinds return the same EvaluationResult structure, so only the reference_text you supply differs.

Reference text

reference_text is the text the speaker is expected to have said: the thing the audio is scored against. This is the field that makes CAPT what it is: the evaluation is anchored to it.

cfg = capt.EvaluationConfig(
    model_id="en_US-16khz",
    reference_text="WHEN THE SUNLIGHT STRIKES",
)

A few properties worth knowing:

  • Lexicon lookup is case-insensitive, so when, When and WHEN behave identically. The uppercase reference text used throughout these examples is a convention, not a requirement.
  • Punctuation handling is a per-model setting, RemovePunctuation in the model’s [Tokenization] config, and it is off in the en_US-16khz model shipped today. Punctuation therefore stays attached to the word: a reference of WHEN, THE SUNLIGHT STRIKES comes back with text of WHEN,, and because the token no longer matches the lexicon it is phonemized by G2P, which changed that word’s score in our own testing. Sentence-final marks are usually harmless, but the safe habit is to send the words alone. See Tuning and Customization.
  • Words not in the lexicon are handled by a G2P model, so invented words, proper nouns and non-words can be used as reference text without any setup. This is what makes non-word decoding tasks work. See Tuning and Customization.
  • Words with several valid pronunciations are all considered, and the variant that best fits the audio is the one reported. A speaker is not penalized for saying “THE” as D i rather than D @.

Alternative reference text

alternative_reference_text accepts additional reference texts that would also be acceptable. Each is aligned independently and returned in alternative_alignments with its own score, and the top-level score becomes the best score across the primary and all alternatives.

cfg = capt.EvaluationConfig(
    model_id="en_US-16khz",
    reference_text="WHEN THE SUNLIGHT STRIKES",
    alternative_reference_text=["WHEN THE SUN LIGHTS STRIKE"],
)

Audio format

audio_format tells the server how to interpret the bytes you send. Two shapes are available.

For files that carry their own header, name the container and let the server read the rest:

cfg = capt.EvaluationConfig(
    model_id="en_US-16khz",
    reference_text="WHEN THE SUNLIGHT STRIKES",
    audio_format=capt.AudioFormat(
        audio_format_headered=capt.AUDIO_FORMAT_HEADERED_WAV,
    ),
)

Supported headered formats are WAV, MP3, FLAC and OGG Opus. Headers are expected once, at the beginning of the stream, not in every Audio message.

audio_format may be omitted entirely for headered audio. The server then detects the format from the header, which is why several examples in these pages leave it out. Naming it explicitly is still recommended: it turns an unreadable or unexpected header into a clear error instead of a detection attempt.

Raw audio is the exception. It carries no header to detect, so audio_format_raw must be supplied in full, and encoding must be set to something other than AUDIO_ENCODING_UNSPECIFIED or the request is rejected.

For raw samples with no header, describe them fully:

cfg = capt.EvaluationConfig(
    model_id="en_US-16khz",
    reference_text="WHEN THE SUNLIGHT STRIKES",
    audio_format=capt.AudioFormat(
        audio_format_raw=capt.AudioFormatRAW(
            encoding=capt.AUDIO_ENCODING_SIGNED,
            bit_depth=16,
            byte_order=capt.BYTE_ORDER_LITTLE_ENDIAN,
            sample_rate=16000,
            channels=1,
        ),
    ),
)

Every server supports raw little-endian 16-bit signed samples at the model’s own sample rate; support for the other formats depends on how the server was built and configured.

Request metadata

metadata is optional and is never used to influence evaluation. The server may record it in logs or stored results.

cfg = capt.EvaluationConfig(
    model_id="en_US-16khz",
    reference_text="WHEN THE SUNLIGHT STRIKES",
    metadata=capt.EvaluationMetadata(
        custom_id="session-42-item-07",
        custom_metadata='{"item":"7","form":"A"}',
    ),
)
  • custom_id accepts up to 64 bytes, restricted to letters, digits, hyphens and underscores. Useful as a tracing or correlation ID.
  • custom_metadata is a free-form string; a plain tag or structured data such as JSON.

Once you have a config, continue to Streaming Evaluation.

3.1.5 - Streaming Evaluation

Describes how to stream audio to CAPT server and receive evaluation results.

CAPT offers two ways to submit audio.

  • StreamingEvaluate is the primary method. It is bidirectional: you send audio as it is captured and receive results as the audio is processed. Use it for anything interactive, and for audio of any length.
  • Evaluate is a unary convenience method that takes the whole audio in a single request and returns a single result. It is intended for short audio, less than a minute, and is the simplest way to score an audio file.

Both accept the same EvaluationConfig and return the same EvaluationResult.

Streaming from an audio file

In a StreamingEvaluate stream, the first message must contain the config, and every subsequent message must contain audio.

An empty Audio message ends the stream. CAPT server treats it as the end of the audio: it finishes processing, sends the final result, and accepts nothing further, so a later send fails. Over gRPC you normally do not need it, because half-closing the stream (CloseSend in Go, exhausting the request iterator in Python) says the same thing. It matters for browser clients, where a websocket has no equivalent half-close.

Do not send one to mean “no audio yet”. If it arrives before the audio is complete, what you get depends on the format, and a client’s state machine needs to handle both:

  • You always receive a final result for the audio sent so far, with is_partial false. It reflects the truncated input, not the utterance you intended, so it will usually be full of deletions.
  • With a headered format the stream then fails. The header promised more data than arrived, so the server reports a container error such as wav: unexpected EOF after the final result. This happens whether or not you send anything afterwards.
  • With raw audio the stream ends cleanly, because there is no container to leave incomplete. The result is still only of the audio you sent.

In other words: a final result is not by itself evidence that the whole utterance was processed. Send the empty message only once the recording is complete, and treat an error after a final result as “the result is incomplete”, not “the result is invalid”.

The examples below stream a WAV file in 100 ms chunks and print each result as it arrives.

import grpc
import cobaltspeech.capt.v1.capt_pb2 as capt
import cobaltspeech.capt.v1.capt_pb2_grpc as capt_grpc

serverAddress = "localhost:2727"

channel = grpc.insecure_channel(serverAddress)
client = capt_grpc.CAPTServiceStub(channel)

# Get the list of models on the server and use the first one.
modelResp = client.ListModels(capt.ListModelsRequest())
modelID = modelResp.models[0].id

cfg = capt.EvaluationConfig(
    model_id=modelID,
    reference_text="WHEN THE SUNLIGHT STRIKES",
    audio_format=capt.AudioFormat(
        audio_format_headered=capt.AUDIO_FORMAT_HEADERED_WAV,
    ),
)

# The first request must contain only the configuration; subsequent
# requests carry audio bytes. A generator is a convenient way to do this.
def stream(cfg, audio, bufferSize=3200):
    yield capt.StreamingEvaluateRequest(config=cfg)

    data = audio.read(bufferSize)
    while len(data) > 0:
        yield capt.StreamingEvaluateRequest(audio=capt.Audio(data=data))
        data = audio.read(bufferSize)

with open("test.wav", "rb") as audio:
    for resp in client.StreamingEvaluate(stream(cfg, audio)):
        if resp.HasField("error"):
            print(f"warning: {resp.error.message}")

        result = resp.evaluation_result
        print(f"partial={result.is_partial} score={result.score:.4f}")

        # Only act on final results; partials will still change.
        if not result.is_partial:
            for word in result.alignments:
                print(f"  {word.text}: " + " ".join(
                    f"{t.reference}={t.score:.2f}" for t in word.tokens
                ))
package main

import (
	"context"
	"fmt"
	"io"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	captpb "github.com/your-org/your-module/gen/go/cobaltspeech/capt/v1"
)

func main() {
	const serverAddress = "localhost:2727"

	audio, err := os.ReadFile("test.wav")
	if err != nil {
		fmt.Printf("failed to read audio: %v\n", err)
		os.Exit(1)
	}

	conn, err := grpc.NewClient(serverAddress,
		grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}
	defer conn.Close()

	client := captpb.NewCAPTServiceClient(conn)

	stream, err := client.StreamingEvaluate(context.Background())
	if err != nil {
		fmt.Printf("failed to open stream: %v\n", err)
		os.Exit(1)
	}

	// The first message must contain the config.
	if err := stream.Send(&captpb.StreamingEvaluateRequest{
		Request: &captpb.StreamingEvaluateRequest_Config{
			Config: &captpb.EvaluationConfig{
				ModelId:       "en_US-16khz",
				ReferenceText: "WHEN THE SUNLIGHT STRIKES",
				AudioFormat: &captpb.AudioFormat{
					AudioFormat: &captpb.AudioFormat_AudioFormatHeadered{
						AudioFormatHeadered: captpb.AudioFormatHeadered_AUDIO_FORMAT_HEADERED_WAV,
					},
				},
			},
		},
	}); err != nil {
		fmt.Printf("failed to send config: %v\n", err)
		os.Exit(1)
	}

	// Send audio in the background while results are read below.
	go func() {
		const chunk = 3200 // 100ms of 16kHz 16-bit mono audio.

		// min() is a builtin from Go 1.21; on older toolchains compute the
		// bound explicitly.
		for i := 0; i < len(audio); i += chunk {
			end := min(i+chunk, len(audio))

			if err := stream.Send(&captpb.StreamingEvaluateRequest{
				Request: &captpb.StreamingEvaluateRequest_Audio{
					Audio: &captpb.Audio{Data: audio[i:end]},
				},
			}); err != nil {
				return
			}
		}

		_ = stream.CloseSend()
	}()

	for {
		resp, err := stream.Recv()
		if err == io.EOF {
			break
		}

		if err != nil {
			fmt.Printf("failed to receive result: %v\n", err)
			os.Exit(1)
		}

		if e := resp.GetError(); e != nil {
			fmt.Printf("warning: %v\n", e.GetMessage())
		}

		result := resp.GetEvaluationResult()
		fmt.Printf("partial=%v score=%.4f\n", result.GetIsPartial(), result.GetScore())

		// Only act on final results; partials will still change.
		if !result.GetIsPartial() {
			for _, word := range result.GetAlignments() {
				fmt.Printf("  %s:", word.GetText())

				for _, t := range word.GetTokens() {
					fmt.Printf(" %s=%.2f", t.GetReference(), t.GetScore())
				}

				fmt.Println()
			}
		}
	}
}

Streaming a 1.5 second recording of “when the sunlight strikes” against that reference produces a short sequence of results:

partial=true  score=0.5514
partial=true  score=0.6128
partial=true  score=0.6128
partial=false score=0.6128

Partial results

While audio is still arriving, the server emits partial results, marked is_partial = true. A partial reflects the audio received so far, and any part of it may change as more audio arrives. In the sequence above the score rises as the last word is heard.

When the stream ends, the server emits a final result with is_partial = false. That result is stable.

  • Use partials for live feedback: highlighting words as they are read, showing a provisional score, driving a progress indicator.
  • Use the final result for any decision you record. Scoring an item on a partial risks scoring it on half a word.

Speech endpointing

A result may set is_speech_endpoint = true to indicate the server believes the speaker has finished, typically after a period of silence. This is useful for closing the microphone automatically once an answer has been given.

Endpoint detection is optional, and clients must be robust to the field never being set. Do not treat it as the signal that results are final; that is what is_partial = false is for.

Non-fatal errors

Both response types carry an optional EvaluationError beside the result. It reports conditions that degrade quality but do not stop processing, most commonly audio sampled at a lower rate than the model expects. The server continues, and you may keep streaming. Log these rather than aborting; they usually point at a recording pipeline that needs attention.

Evaluating a whole file at once

For short audio, Evaluate avoids the streaming machinery entirely. Over the HTTP+JSON gateway that is a single POST:

curl -s -X POST http://localhost:8080/api/capt/v1/evaluate \
  -H 'Content-Type: application/json' \
  -d '{
    "config": {
      "model_id": "en_US-16khz",
      "reference_text": "WHEN THE SUNLIGHT STRIKES",
      "audio_format": {"audio_format_headered": "AUDIO_FORMAT_HEADERED_WAV"}
    },
    "audio": {"data": "<base64-encoded WAV file>"}
  }'

The data field is the audio file, base64 encoded. See Interpreting Results for what comes back.

3.1.6 - Interpreting Results

How to read an EvaluationResult and turn it into a decision.

This is the page worth reading carefully. Getting audio into CAPT is straightforward; deciding what its output means for your application is where the design work is.

The shape of a result

An EvaluationResult is a tree:

EvaluationResult
├── score                    overall score for the utterance, 0..1
├── is_partial               true while more audio may still change this
├── is_speech_endpoint       true if the speaker appears to have finished
├── alignments               [ ]AlignedWord
│   └── AlignedWord
│       ├── text             the reference word
│       ├── start_time_ms    when the word starts in the audio
│       ├── duration_ms      how long it lasts
│       └── tokens           [ ]AlignedToken   (the phonemes)
│           └── AlignedToken
│               ├── kind           MATCH | SUBSTITUTION | DELETION | INSERTION
│               ├── reference      the expected phoneme
│               ├── score          0..1 for this phoneme
│               └── hypotheses     [ ]AlignmentHypothesis  (what was heard)
│                   └── token, confidence, start_time_ms, duration_ms
└── alternative_alignments   the same, once per alternative reference text

The reference text drives the structure: your words appear as AlignedWord entries in order, and within each one there is an AlignedToken per expected phoneme.

The four alignment kinds

Every expected phoneme gets exactly one of four verdicts.

KindMeaningreferencehypothesesscore
MATCHThe expected phoneme was heard with enough confidencesetpopulated0.5 to 1.0
SUBSTITUTIONThe expected phoneme was not heard with enough confidence: either something else was heard in its place, or the phoneme itself was heard but below its cutoffsetpopulated0 to below 0.5
DELETIONThe phoneme was not heard at allsetempty0
INSERTIONSomething extra was heard that the reference does not account foremptypopulated0

A real example makes this concrete. Below is the result of scoring a recording of “when the sunlight strikes” against a deliberately wrong reference, “WHEN THE MOONLIGHT STRIKES” (phonemes are X-SAMPA):

overall score: 0.5157

WHEN         MATCH w (1.00)  MATCH E (0.99)  MATCH n (1.00)
THE          MATCH D (0.87)  MATCH @ (0.67)
MOONLIGHT    SUB   m (0.00)  SUB   u (0.00)  MATCH n (0.65)
             MATCH l (0.78)  MATCH aI (0.82) MATCH t (0.73)
STRIKES      MATCH s (0.79)  DEL   t (0.00)  DEL   r\ (0.00)
             DEL   aI (0.00) SUB   k (0.49)  DEL   s (0.00)

Read that as a diagnosis, not just a number. The first two words were said as expected. In MOONLIGHT the leading m u was not there (the speaker said s V n), but the shared tail n l aI t matched, so those phonemes score well. STRIKES shows what happens when the audio has already run out: the reference still has phonemes to account for, and they come back as deletions.

The overall score of 0.52 is the average of that mixture. This is why an overall score alone is rarely the right thing to threshold. 0.52 here does not mean “roughly half right”, it means “two words right and two words wrong”.

Insertions

Insertions do not belong to any reference word, so they are reported in their own AlignedWord entries with an empty text, positioned in the sequence where they were heard. Scoring the same audio against the single word “ELEPHANT” shows this:

overall score: 0.3966

(insertion)  INS  (heard: w)
ELEPHANT     MATCH E (0.99)  SUB l (0.00)  MATCH @ (0.67)
             SUB   f (0.00)  SUB n= (0.00) MATCH t (0.73)
(insertion)  INS  (heard: k)  INS (heard: s)

The speaker said far more than the reference accounted for, and the leftover audio at each end is reported as insertions rather than being silently discarded.

Where the numbers come from

Phoneme scores

The acoustic model does not commit to a single phoneme per time slice. It produces a distribution, a confusion network, and the score for an expected phoneme is derived from how much probability mass landed on it.

You can see this in the hypotheses list. Here is a single MATCH from a real result:

{
  "kind": "ALIGNMENT_KIND_MATCH",
  "reference": "n",
  "score": 0.9798913,
  "hypotheses": [
    { "token": "n",  "confidence": 0.963, "start_time_ms": "570", "duration_ms": "119" },
    { "token": "m",  "confidence": 0.029, "start_time_ms": "570", "duration_ms": "119" },
    { "token": "n=", "confidence": 0.006, "start_time_ms": "570", "duration_ms": "119" },
    { "token": "N",  "confidence": 0.002, "start_time_ms": "570", "duration_ms": "119" }
  ]
}

The model heard n with 0.963 confidence, but also considered m, n= and N. The reported score of 0.98 is that confidence remapped through the model’s cutoff for n. See Tuning and Customization for how that remapping works and how to change it.

The practical consequence: a score is a calibrated quantity, not a raw probability. A cutoff is chosen so that a score of exactly 0.5 sits at the boundary between acceptable and unacceptable for that phoneme. That is what makes 0.5 a meaningful place to threshold, and it is why different phonemes can be held to different standards.

Word and utterance scores

An AlignedWord does not carry its own score field; a word’s quality is the scores of its phonemes. Compute whatever summary suits your task: the mean, the minimum, or the fraction of phonemes that came back MATCH.

The top-level score is the overall score for the utterance: the mean of the scores of the reference tokens. If alternative reference texts were supplied, it is the best score across the primary and all alternatives.

Timestamps

Timestamps live on the hypotheses, not on the token. An AlignedToken has no time fields of its own; start_time_ms and duration_ms are on each AlignmentHypothesis inside it, and on the enclosing AlignedWord.

This has one consequence that catches people out:

Likewise, a word made up entirely of deletions has nothing to anchor to, and its start_time_ms and duration_ms are reported as 0.

Turning results into a decision

A pass/fail mark for one item

If your application needs a yes/no verdict (did the speaker say the target correctly?), do not reach for the overall score first. Consider what the task actually requires:

  • Strict, verbatim tasks. Any divergence is a failure. Require every token to be MATCH:

    correct = all(
        t.kind == capt.ALIGNMENT_KIND_MATCH
        for w in result.alignments
        for t in w.tokens
    )
    

    This is exactly right when the rule is “any change at all, however minor, is wrong”, because substitutions, deletions and insertions all break it.

  • Tolerant tasks. Some divergence is acceptable. A slightly indistinct consonant should not fail an otherwise good attempt. Threshold on the proportion of matched phonemes, or on the mean phoneme score, rather than demanding a clean sweep. Note that a tolerant rule built on the overall score, or on reference tokens alone, will not notice extra speech: decide separately whether insertions should fail the item.

  • Targeted tasks. Only some phonemes matter: the contrast the item is testing. Score only those tokens and ignore the rest.

Choosing between several candidate answers

When you have a known correct answer and a set of known incorrect answers, and you need to know which one was said, evaluate the audio once per candidate and compare the resulting scores:

candidates = ["BED", "LOUNGE", "SOFA"]
scores = {}

for candidate in candidates:
    cfg = capt.EvaluationConfig(model_id=modelID, reference_text=candidate)
    with open(path, "rb") as audio:
        for resp in client.StreamingEvaluate(stream(cfg, audio)):
            if not resp.evaluation_result.is_partial:
                scores[candidate] = resp.evaluation_result.score

best = max(scores, key=scores.get)

This gives you a score per candidate, so you can see not just the winner but the margin, so you can reject the result as unclear when two candidates score alike. Packing the candidates into alternative_reference_text instead would collapse them into a single best score and lose exactly that information.

Calibrating against human judgement

If you are automating a decision a person currently makes, the metric that matters is agreement with that person, not any internal notion of accuracy.

The recommended approach is to collect audio alongside the human verdict, run CAPT over it, then sweep your decision rule across the collected set and count true positives, true negatives, false positives and false negatives at each setting. That tells you where to set the threshold, and, just as importantly, what the residual disagreement rate is and which way it leans. A rule that is wrong in the safe direction for your use case is often better than one that is wrong less often overall.

Per-phoneme cutoffs can then correct systematic biases that a single global threshold cannot; see Tuning and Customization.

Alternative alignments

If you supplied alternative_reference_text, each alternative comes back in alternative_alignments as an AlternativeAlignment carrying its own reference_text, score, and full alignments tree with the same structure described above.

The top-level score is the best across the primary and the alternatives, so if you only care whether any acceptable rendering was produced, the top-level score is enough. If you care which one, see above.

3.1.7 - Phoneme Models

Evaluating individual phonemes produced in isolation.

CAPT models come in two kinds, reported as kind by ListModels.

  • Word models (MODEL_KIND_SPEECH_EVALUATION) evaluate phonemes in word context. The reference_text is ordinary text, and this is what you want for reading words, sentences and passages.
  • Phoneme models (MODEL_KIND_PHONEME_EVALUATION) evaluate phonemes produced in isolation. The reference_text is a sequence of phones.

The distinction matters because a sound produced on its own is acoustically quite different from the same sound inside a word. A model trained on connected speech often mis-recognizes an isolated phoneme, because nothing in its training looked like that. If your task asks a speaker to produce a single sound (“say the /sh/ sound”), a phoneme model is substantially more reliable.

Both kinds are used through exactly the same API calls and return the same EvaluationResult. Only the reference_text differs.

Reference text for phoneme models

For a phoneme model, reference_text is one or more phones separated by whitespace:

# A single phone.
cfg = capt.EvaluationConfig(model_id=phonemeModelID, reference_text="AA")

# A sequence of phones.
cfg = capt.EvaluationConfig(model_id=phonemeModelID, reference_text="Z AE P")

Phone groups

Some phones are only meaningfully produced as a pair, and the model treats such a pair as a single unit called a phone group. Phone groups are written with the members joined by a period:

# A phone group.
cfg = capt.EvaluationConfig(model_id=phonemeModelID, reference_text="AO.NG")

# Mixed with ordinary phones.
cfg = capt.EvaluationConfig(model_id=phonemeModelID, reference_text="F AE.NG")

For all practical purposes a phone group behaves as its own distinct phone: it is matched, substituted or deleted as a single token, and receives a single score.

The phone set

The phones a model accepts are specific to that model, both the inventory and which groups exist. Supplying a phone the model does not know is an error, so check against the set for the model you were given. A typical US English phoneme model accepts:

AA     AA.R   AE     AE.NG  AH     AH.NG  AO     AO.NG
AO.R   AW     AY     B      CH     D      DH     EH
EH.R   ER     EY     F      G      HH     IH     IH.NG
IH.R   IY     JH     K      L      M      N      NG
OW     OY     P      R      S      SH     T      TH
UH     UW     V      W      Y      Y.UW   Z      ZH

Reading the results

Results have the same structure as any other evaluation. For each phone in the reference_text there is an aligned token:

  • ALIGNMENT_KIND_MATCH means the phone was found in the audio with a high degree of confidence.
  • ALIGNMENT_KIND_SUBSTITUTION means a different phone was recognized in its place, or the phone was recognized only with low confidence.
  • ALIGNMENT_KIND_DELETION means the phone was not recognized at all.

Anything else recognized in the audio that does not align to the reference is reported as insertions, grouped separately.

Isolated-phoneme audio very often contains more than the phoneme itself: a speaker clearing their throat, a lead-in, or the assessor’s prompt. Those show up as insertions, which is why the example below scores 1.0 despite three extra tokens: the reference phone ZH was produced correctly, and the surrounding material is reported rather than being allowed to affect the score of the phone you asked about.

{
  "score": 1.0,
  "alignments": [
    {
      "start_time_ms": "680",
      "duration_ms": "360",
      "tokens": [
        {
          "kind": "ALIGNMENT_KIND_INSERTION",
          "hypotheses": [
            { "token": "T", "confidence": 1.0, "start_time_ms": "680", "duration_ms": "120" }
          ]
        },
        {
          "kind": "ALIGNMENT_KIND_INSERTION",
          "hypotheses": [
            { "token": "R", "confidence": 1.0, "start_time_ms": "840", "duration_ms": "120" }
          ]
        },
        {
          "kind": "ALIGNMENT_KIND_INSERTION",
          "hypotheses": [
            { "token": "EH", "confidence": 0.994, "start_time_ms": "960", "duration_ms": "80" }
          ]
        }
      ]
    },
    {
      "text": "ZH",
      "start_time_ms": "1000",
      "duration_ms": "240",
      "tokens": [
        {
          "kind": "ALIGNMENT_KIND_MATCH",
          "reference": "ZH",
          "score": 1.0,
          "hypotheses": [
            { "token": "ZH", "confidence": 1.0, "start_time_ms": "1000", "duration_ms": "240" }
          ]
        }
      ]
    }
  ]
}

If extraneous audio should count against the speaker in your task, inspect the insertion entries and apply your own rule; the information is there either way.

3.1.8 - Tuning and Customization

Pronunciations, per-phoneme thresholds and scoring behavior.

A general-purpose speech model is a fixed object: you send it audio and accept what it returns. CAPT is deliberately not that. Because the scoring stage is separate from the acoustic model, a great deal of behavior can be adjusted for your task without retraining anything, by changing which pronunciations count as correct, and how strictly each sound is judged.

Everything on this page lives in the model directory shipped with your release, and takes effect when the server restarts.

models/en_US-16khz/capt/
├── model.config.toml     scoring behavior and paths to the files below
├── lexicon.json          the primary pronunciation dictionary
├── lexicon_addenda.tsv   your additions and overrides
├── cutoffs.yaml          per-phoneme thresholds
└── g2p.ort               grapheme-to-phoneme model for unknown words

The model config points at each of these by path, so the names above are the convention rather than a requirement. Older model directories may carry the cutoffs as a plain-text cutoffs.txt instead; the meaning is identical and CutoffsPath says which file is in use.

Pronunciations

The lexicon and its addenda

The lexicon maps words to their phoneme sequences. A word may have several valid pronunciations, and CAPT considers all of them, reporting whichever best fits the audio, so a speaker is not penalized for a legitimate variant.

lexicon_addenda.tsv is loaded after the primary lexicon and is the file you should edit. Entries in it add new words and override existing ones, leaving the shipped lexicon intact so that upgrades stay clean.

This is the mechanism to reach for when your task defines what counts as an acceptable pronunciation. If an item specifies that a target may be produced two ways, add both and each will be accepted as a match:

MOSP	m A s p
MOSP	m oU s p

Those two entries accept mosp pronounced to rhyme with wasp, or with soap.

Out-of-vocabulary words and G2P

Words that are in neither the lexicon nor the addenda are passed to a grapheme-to-phoneme model, which predicts a pronunciation from spelling. This happens automatically and requires no configuration.

It means invented words, proper nouns and non-words can be used as reference_text with no setup at all, which is useful for decoding tasks built on pseudo-words, where the whole point is that the string is not a real word.

Tokenization

[Tokenization] in model.config.toml controls how reference text is pre-processed:

SettingWhat it doesDefaultIn en_US-16khz
RemovePunctuationStrips punctuation before lexicon lookup. With it off, punctuation stays attached to the word and is phonemized as part of it.offoff
UseSentenceTokenizerSplits a long reference into sentences before tokenizing.offon
MaxInputLengthSafeguard on the length of a reference text. Counted in characters, not words; a longer reference is rejected with an error. Applies to each alternative reference text too.20482048

“Default” is what applies when the key is absent from model.config.toml. Neither boolean has a server-side default beyond false, so what matters in practice is what your model ships with.

Cutoffs and acceptable confusions

This is the most useful tuning mechanism CAPT offers, and the least obvious.

Raw acoustic confidences are biased in ways that are specific to a model and a population of speakers. A model may routinely blur m and n; young children may produce th in a way that reads as s. Left alone, those biases show up as wrong verdicts. Cutoffs correct them by remapping confidence to score.

Each symbol has a cutoff, the confidence at which it is exactly borderline. The remapping is piecewise linear and pins that point to a score of 0.5:

  • raw confidence in [0, cutoff] maps to a score in [0, 0.5]
  • raw confidence in [cutoff, 1] maps to a score in [0.5, 1]

So the cutoff is the dial for how strict a given phoneme is. Lowering it is more lenient; raising it demands higher confidence before the phoneme counts as correct. Any symbol not listed uses DefaultCutoffScore from the [Scoring] section.

A modifier extends this with acceptable confusions. It names alternatives (other phonemes that may also count toward the match) and a multiplier that scales the cutoff when one of them is what was actually heard:

"A":
  cutoff: 0.50
  modifier:
    multiplier: 1.50
    alternatives: ["O"]

"T":
  cutoff: 0.50
  modifier:
    multiplier: 1.90
    alternatives: ["s"]

Read the first entry as: "A (the vowel of lot) needs 50% confidence. O (the vowel of thought) is close enough to count, but only if the model is much more sure of it: 0.50 × 1.50 = 0.75." The second is stricter still: s may stand in for T (th), but must clear 0.50 × 1.90 = 0.95, which is about right for a contrast that many young speakers have not yet acquired.

During alignment the confidences of the reference phoneme and any listed alternatives are summed, and the scaled cutoff is applied to that sum.

The effect is a per-sound tolerance you control. Rather than one global threshold that is too strict for some phonemes and too lax for others, you set the bar sound by sound, and you do it by editing a YAML file, not by collecting data and retraining.

Scoring behavior

The [Scoring] section of model.config.toml governs the alignment itself. The settings most likely to matter:

SettingWhat it does
DefaultCutoffScoreThe cutoff for any symbol not listed in cutoffs.yaml. Below 0.5 is lenient, above 0.5 is strict.
SubCost, InsCost, DelCostEdit-distance costs for substitution, insertion and deletion. Raising one makes the aligner prefer explanations that avoid it.
AlignVowelsPrevents nonsensical vowel↔consonant substitutions, reporting a deletion plus an insertion instead.
IncludeStressTreat stress markers as distinguishing (AH0 ≠ AH1). Requires both the lexicon and the acoustic model to carry stress.
IncludeIntraWordInsertionWhether extra phonemes heard inside a word are attached to that word or reported separately.
NonSpeechTokenConfidenceScalingWeights the confidence of silence and noise tokens. Below 1.0 makes the model less willing to call something non-speech, which is useful in noisy rooms.
CnetLinkThresholdDiscards time slices where the acoustic model put little probability on any speech token.
MaxCandidatePronsCaps how many combinations of word pronunciations are enumerated for a reference.

Non-speech tokens

NonSpeechTokens in [Vocab] lists the symbols the acoustic model emits for things that are not speech: silence, breath, vocal noise, unknown sounds. These are handled specially: they are excluded from scoring rather than being treated as mispronunciations, and they interact with CnetLinkThreshold and NonSpeechTokenConfidenceScaling above.

Recordings made in real rooms contain plenty of this material. If you find that noisy recordings score too harshly, these settings, rather than the cutoffs, are usually the right place to look.

Adapting the acoustic model

Configuration handles a great deal, but not everything. Where a population is genuinely different from what the model was trained on (young children, a regional accent, a specific recording setup), the acoustic model itself can be adapted to it, which addresses the cause rather than compensating for it downstream.

Adaptation needs representative audio, ideally paired with the verdicts you want the system to reproduce. If your application already records both, you may have a suitable dataset without any additional collection effort. Contact us to discuss what your data supports.

3.1.9 - Browser Integration

Calling CAPT from a web browser over HTTP and websockets.

Native applications should use gRPC directly, as shown throughout these docs. Browsers cannot open a bidirectional gRPC stream, so CAPT server also exposes an HTTP+JSON gateway and a websocket endpoint for StreamingEvaluate.

The HTTP API is not a property of the server itself: it is served only when api.Address is set. The configuration shipped with a release sets it to :8080, which is why the getting started page can curl it straight away. If you are writing a config from scratch, enable it explicitly in capt-server.cfg.toml:

[server.http]
api.Address = ":8080"

## Serve the built-in web demo alongside the API.
api.EnableWebDemo = true

## Only needed when the page is served from a different origin
## than the API, e.g. during local development.
# api.EnableCORS = true

JSON conventions

The gateway emits JSON using protobuf field names, so:

  • Field names are snake_case, exactly as in the proto: evaluation_result, start_time_ms, alternative_reference_text.
  • Enums are full strings, not numbers: "ALIGNMENT_KIND_MATCH", "AUDIO_FORMAT_HEADERED_WAV".
  • Unpopulated fields are emitted, so a field being present does not imply it was set.
  • 64-bit integers are strings, per the protobuf JSON mapping: "start_time_ms": "570", not 570. Parse these before doing arithmetic.
  • bytes fields are base64, so audio is base64-encoded into {"audio": {"data": "..."}}.

Unary calls over HTTP

The two metadata calls and Evaluate are available as ordinary HTTP requests:

MethodRoute
VersionGET /api/capt/v1/version
ListModelsGET /api/capt/v1/list-models
EvaluatePOST /api/capt/v1/evaluate
StreamingEvaluateGET /api/capt/v1/streaming-evaluate (websocket)
const models = await fetch('/api/capt/v1/list-models').then((r) => r.json());
const modelID = models.models[0].id;

Streaming over a websocket

StreamingEvaluate is served over a websocket that carries the same messages as the gRPC stream, one JSON object per websocket message.

The sequence mirrors the gRPC one: the first message must be the config, and every message after it carries audio.

const ws = new WebSocket(`wss://${location.host}/api/capt/v1/streaming-evaluate`);

ws.onopen = () => {
  // First message: the configuration.
  ws.send(
    JSON.stringify({
      config: {
        model_id: 'en_US-16khz',
        reference_text: 'WHEN THE SUNLIGHT STRIKES',
        audio_format: { audio_format_headered: 'AUDIO_FORMAT_HEADERED_WAV' }
      }
    })
  );
};

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data);

  // The gateway reports a stream-level failure at the top level; a non-fatal
  // EvaluationError is a field of the response, so it sits inside the envelope.
  if (msg.error != null) {
    console.error('stream failed:', msg.error);
    return;
  }

  if (msg.result?.error != null) {
    console.warn('non-fatal:', msg.result.error.message);
  }

  const result = msg.result?.evaluation_result;
  if (result == null) {
    return;
  }

  if (result.is_partial) {
    showProvisional(result); // live highlighting
  } else {
    recordFinal(result); // the result you act on
  }
};

Sending audio means base64-encoding each chunk:

// Spreading a whole buffer into String.fromCharCode overflows the call stack
// on anything but small inputs, so encode in fixed-size blocks.
function toBase64(buf) {
  const bytes = new Uint8Array(buf);
  const block = 0x8000;
  let binary = '';

  for (let i = 0; i < bytes.length; i += block) {
    binary += String.fromCharCode.apply(null, bytes.subarray(i, i + block));
  }

  return btoa(binary);
}

async function sendChunk(blob) {
  ws.send(JSON.stringify({ audio: { data: toBase64(await blob.arrayBuffer()) } }));
}

// A websocket has no half-close, so an empty audio message is how you say
// "that is all the audio". Send it only once the recording is complete.
function endStream() {
  ws.send(JSON.stringify({ audio: { data: '' } }));
}

The same helper is what you want for the whole-recording POST described below; a complete take is far past the size at which the one-line spread form fails.

Capturing audio in the browser

The audio you send must match the audio_format you declared. The MediaRecorder API defaults to a compressed format that CAPT does not accept, so a browser client generally needs an encoder that can emit WAV chunks while recording, rather than only at the end.

Two constraints are worth designing around:

  • Send small, regular chunks. Around 100-250 ms keeps partial results flowing smoothly. Larger chunks make feedback feel laggy.
  • Declare the header once. For headered formats the header belongs at the start of the stream, not on every chunk.

If you do not need live feedback, it is considerably simpler to record the whole utterance, then POST it to /api/capt/v1/evaluate as a single base64 blob.

The built-in demo

With api.EnableWebDemo = true, the server serves a demo application at the HTTP address, by default http://localhost:8080. It records from the microphone or accepts an uploaded file, streams it over the websocket described above, and renders the per-word and per-phoneme results.

It is a useful way to sanity-check a server, a model and a piece of audio before writing any client code.

3.1.10 - API Reference

Detailed reference for API requests and types.

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

CAPTService

Service that implements the Cobalt CAPT API.

Version

Version(VersionRequest) VersionResponse

Returns version information from the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

Returns information about the models available on the server.

Evaluate

Evaluate(EvaluateRequest) EvaluateResponse

Performs synchronous speech evaluation by receiving results after all audio has been sent and processed. It is expected that this request be typically used for short audio content: less than a minute long. For longer content, the StreamingEvaluate method should be preferred.

StreamingEvaluate

StreamingEvaluate(StreamingEvaluateRequest) StreamingEvaluateResponse

Performs bidirectional streaming for evaluating speech by receiving results while sending audio. This method is only available via GRPC and not via HTTP+JSON. However, a web browser may use websockets to use this service.

Messages

  • If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
  • If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

AlignedToken

AlignedToken contains a single reference token and the alternate hypotheses recognized from the audio that could be aligned with the reference token.

If kind == ALIGNMENT_KIND_MATCH or ALIGNMENT_KIND_SUBSTITUTION, both the reference token and hypotheses list will be populated. The score will be based on the confidence value of the reference token within the hypotheses. In the case of a substitution, the hypotheses list may not contain the actual reference token at all, and the score will be 0.

If kind == ALIGNMENT_KIND_DELETION, the hypotheses will be an empty list and score will be 0.

If kind == ALIGNMENT_KIND_INSERTION, the reference token will be an empty string and score will be 0.

Fields

  • kind (AlignmentKind ) The kind of alignment for this token.

  • reference (string ) The actual reference token.

  • score (float ) The score for this token, between 0 and 1 inclusive, based on the confidence value of the token within the recognized token hypotheses from the audio.

  • hypotheses (AlignmentHypothesis repeated) The recognized tokens from the audio that have been aligned to this reference token.

AlignedWord

AlignedWord contains the aligned tokens within a single word in the reference text.

Fields

  • text (string ) The actual word.

  • start_time_ms (uint64 ) The timestamp at which this token starts in the audio, in milliseconds.

  • duration_ms (uint64 ) The duration of this token in the audio, in milliseconds.

  • tokens (AlignedToken repeated) The tokens that make up the word.

AlignmentHypothesis

AlignmentHypothesis contains the metadata for a token recognized from audio, including its timestamps and confidence scores.

Fields

  • token (string ) The actual token.

  • confidence (float ) The confidence with which this token was recognized in the audio, between 0 and 1 inclusive.

  • start_time_ms (uint64 ) The timestamp at which this token starts in the audio, in milliseconds.

  • duration_ms (uint64 ) The duration of this token in the audio, in milliseconds.

AlternativeAlignment

AlternativeAlignment contains alignments against alternative reference text(s) if specified in the EvaluationConfig.

Fields

  • reference_text (string ) The alternative reference text.

  • score (float ) Evaluation score, between 0 and 1 inclusive, with 1 indicating a perfect match between the alternative reference text and what was said in the audio.

  • alignments (AlignedWord repeated) Alignment between the alternative reference text tokens and recognized tokens.

Audio

Audio to be sent to Capt.

Fields

AudioFormat

Format of the audio to be sent for recognition.

Depending on how they are configured, server instances of this service may not support all the formats provided in the API. One format that is guaranteed to be supported is the RAW format with little-endian 16-bit signed samples with the sample rate matching that of the model being requested.

Fields

  • oneof audio_format.audio_format_raw (AudioFormatRAW ) Audio is raw data without any headers

  • oneof audio_format.audio_format_headered (AudioFormatHeadered ) Audio has a self-describing header. Headers are expected to be sent at the beginning of the entire audio file/stream, and not in every Audio message.

    The default value of this type is AUDIO_FORMAT_HEADERED_UNSPECIFIED. If this value is used, the server may attempt to detect the format of the audio. However, it is recommended that the exact format be specified.

AudioFormatRAW

Details of audio in raw format

Fields

  • encoding (AudioEncoding ) Encoding of the samples. It must be specified explicitly and using the default value of AUDIO_ENCODING_UNSPECIFIED will result in an error.

  • bit_depth (uint32 ) Bit depth of each sample (e.g. 8, 16, 24, 32, etc.). This is a required field.

  • byte_order (ByteOrder ) Byte order of the samples. This field must be set to a value other than BYTE_ORDER_UNSPECIFIED when the bit_depth is greater than 8.

  • sample_rate (uint32 ) Sampling rate in Hz. This is a required field.

  • channels (uint32 ) Number of channels present in the audio. E.g.: 1 (mono), 2 (stereo), etc. This is a required field.

EvaluateRequest

The top-level message sent by the client for the Evaluate method. Both the EvaluationConfig and Audio fields are required. The entire audio data must be sent in one request. If your audio data is larger, please use the StreamingEvaluate call.

Fields

EvaluateResponse

The message returned by the server for the Evaluate method.

Fields

  • evaluation_result (EvaluationResult ) Response from the server. The kind of result depends on the kind of the model that was chosen in the EvaluationConfig.

  • error (EvaluationError ) A non-fatal error message. If a server encountered a non-fatal error when processing the request, it will be returned in this message. The server will continue to process audio and produce further results. Clients can continue streaming audio even after receiving these messages. This error message is meant to be informational.

    An example of when these errors maybe produced: audio is sampled at a lower rate than expected by model, producing possibly less accurate results.

    This field will be unset if there is no error to report.

EvaluationConfig

Configuration for a StreamingEvaluateRequest.

Fields

  • model_id (string ) ID of the model to use. A list of supported IDs can be found using the ListModels call.

  • audio_format (AudioFormat ) Format of the audio to be sent.

  • reference_text (string ) Reference Text that is expected to be contained in the audio.

  • alternative_reference_text (string repeated) Alternative reference text(s) that are also acceptable if recognized in the audio.

  • metadata (EvaluationMetadata ) This is an optional field. If there is any metadata associated with the audio being sent, use this field to provide it to the recognizer. The server may record this metadata when processing the request. The server does not use this field for any other purpose.

EvaluationError

Developer-facing error message about a non-fatal process issue.

Fields

EvaluationMetadata

Metadata associated with the evaluation request

Fields

  • custom_metadata (string ) Any custom metadata that the client wants to associate with the recording. This could be a simple string (e.g. a tracing ID) or structured data (e.g. JSON).

  • custom_id (string ) This is an optional field to specify custom ID to identify the evaluation request. The custom ID must be a string of upto 64 bytes, and only alphabets, digits, hyphens and underscores are allowed. This ID may be recorded by the server in logs or other storage, and should therefore not include any sensitive information.

EvaluationResult

EvaluationResult contains the result generated by a speech evaluation model.

Fields

  • is_partial (bool ) If this is set to true, it denotes that the result is an interim partial result, and could change after more audio is processed. If unset, or set to false, it denotes that this is a final result and will not change.

    Servers are not required to implement support for returning partial results, and clients should generally not depend on their availability.

  • score (float ) Overall evaluation score, between 0 and 1 inclusive, with 1 indicating a perfect match between the reference text and what was said in the audio. If alternative reference text(s) are specified, then the score will be take those into account and be based on the reference text that aligns the best with recognized tokens.

  • alignments (AlignedWord repeated) Alignment between the primary expected reference text tokens and recognized tokens.

  • alternative_alignments (AlternativeAlignment repeated) Alignments against alternative reference text tokens (if specified) and recognized tokens.

  • is_speech_endpoint (bool ) If true, indicates that a speech endpoint has been detected.

    A speech endpoint signifies that the server believes the user has finished their utterance, typically after detecting a specific duration of silence.

    NOTE: Server support for endpoint detection is optional. Clients must be robust to this field never being set.

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

  • models (Model repeated) List of models available for use that match the request.

Model

Description of a CAPT model.

Fields

  • id (string ) Unique identifier of the model. This identifier is used to choose the model the model when configuring a StreamingEvaluate request.

  • name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their task.

  • kind (ModelKind ) The specific kind of model. This determines what type of results it sends back, and additional config requirements if any.

  • attributes (ModelAttributes ) Model Attributes.

ModelAttributes

Attributes of a Capt model.

Fields

  • sample_rate (uint32 ) Audio sample rate (native) supported by the model.

  • metadata (ModelMetadata ) Metadata associated with the model.

ModelMetadata

Metadata associated with a Capt model.

Fields

  • version (string ) Model version in semver format (e.g. 1.0.0). If the version is not known, it will default to “unknown”.

  • build_date (string ) Date the model was built in YYYY-MM-DD format. If the version is not known, it will default to “unknown”.

StreamingEvaluateRequest

The top level messages sent by the client for the StreamingEvaluate method. In this streaming call, multiple StreamingEvaluateRequest messages should be sent. The first message must contain a EvaluationConfig message, and all subsequent messages must contain Audio only. All Audio messages must contain non-empty audio. If audio content is empty, the server may choose to interpret it as end of stream and stop accepting any further messages.

Fields

StreamingEvaluateResponse

The message returned by the server for the StreamingEvaluate method.

Fields

  • evaluation_result (EvaluationResult ) Response from the server. The kind of result depends on the kind of the model that was chosen in the EvaluationConfig.

  • error (EvaluationError ) A non-fatal error message. If a server encountered a non-fatal error when processing the request, it will be returned in this message. The server will continue to process audio and produce further results. Clients can continue streaming audio even after receiving these messages. This error message is meant to be informational.

    An example of when these errors maybe produced: audio is sampled at a lower rate than expected by model, producing possibly less accurate results.

    This field will be unset if there is no error to report.

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The message sent by the server for the Version method.

Fields

  • version (string ) Version of the server handling these requests.

Enums

AlignmentKind

AlignmentKind represents one of four alignment outcomes possible: a Match, Substitution, Deletion or Insertion.

NameNumberDescription
ALIGNMENT_KIND_UNSPECIFIED0Default value of this type.
ALIGNMENT_KIND_MATCH1A match implies that the reference and recognized token from audio are a match with a reasonable amount of confidence.
ALIGNMENT_KIND_SUBSTITUTION2A substitution implies that the reference token has been replaced by a different token in the recognized tokens from the audio.
ALIGNMENT_KIND_DELETION3A deletion implies that the reference token was not found in the recognized tokens from audio.
ALIGNMENT_KIND_INSERTION4A insertion implies that a extraneous token has been recognized in the audio, that cannot be matched to any token in the reference.

AudioEncoding

The encoding of the audio data to be sent for recognition.

NameNumberDescription
AUDIO_ENCODING_UNSPECIFIED0AUDIO_ENCODING_UNSPECIFIED is the default value of this type and will result in an error.
AUDIO_ENCODING_SIGNED1PCM signed-integer
AUDIO_ENCODING_UNSIGNED2PCM unsigned-integer
AUDIO_ENCODING_IEEE_FLOAT3PCM IEEE-Float
AUDIO_ENCODING_ULAW4G.711 mu-law
AUDIO_ENCODING_ALAW5G.711 a-law

AudioFormatHeadered

NameNumberDescription
AUDIO_FORMAT_HEADERED_UNSPECIFIED0AUDIO_FORMAT_HEADERED_UNSPECIFIED is the default value of this type.
AUDIO_FORMAT_HEADERED_WAV1WAV with RIFF headers
AUDIO_FORMAT_HEADERED_MP32MP3 format with a valid frame header at the beginning of data
AUDIO_FORMAT_HEADERED_FLAC3FLAC format
AUDIO_FORMAT_HEADERED_OGG_OPUS4Opus format with OGG header

ByteOrder

Byte order of multi-byte data

NameNumberDescription
BYTE_ORDER_UNSPECIFIED0BYTE_ORDER_UNSPECIFIED is the default value of this type.
BYTE_ORDER_LITTLE_ENDIAN1Little Endian byte order
BYTE_ORDER_BIG_ENDIAN2Big Endian byte order

ModelKind

NameNumberDescription
MODEL_KIND_UNSPECIFIED0Default value of this type.
MODEL_KIND_SPEECH_EVALUATION1Model for evaluating the accuracy of spoken text from audio. This is done by first recognizing what’s said in the audio, and then aligning expected and recognized tokens (phonemes, syllables, etc.). This model returns results in the form of EvaluationResult messages.
MODEL_KIND_PHONEME_EVALUATION2Model for evaluating the accuracy of phonemes pronounced in isolation from audio. This is done in a way similar to speech evaluation models, but is more accurate for single phonemes in isolation, which a regular speech model may not recognize correctly. This model returns results in the form of EvaluationResult messages.

Scalar Value Types

.proto TypeC++ TypeC# TypeGo TypeJava TypePHP TypePython TypeRuby Type

double
doubledoublefloat64doublefloatfloatFloat

float
floatfloatfloat32floatfloatfloatFloat

int32
int32intint32intintegerintBignum or Fixnum (as required)

int64
int64longint64longinteger/stringint/longBignum

uint32
uint32uintuint32intintegerint/longBignum or Fixnum (as required)

uint64
uint64ulonguint64longinteger/stringint/longBignum or Fixnum (as required)

sint32
int32intint32intintegerintBignum or Fixnum (as required)

sint64
int64longint64longinteger/stringint/longBignum

fixed32
uint32uintuint32intintegerintBignum or Fixnum (as required)

fixed64
uint64ulonguint64longinteger/stringint/longBignum

sfixed32
int32intint32intintegerintBignum or Fixnum (as required)

sfixed64
int64longint64longinteger/stringint/longBignum

bool
boolboolboolbooleanbooleanbooleanTrueClass/FalseClass

string
stringstringstringStringstringstr/unicodeString (UTF-8)

bytes
stringByteString[]byteByteStringstringstrString (ASCII-8BIT)

3.1.11 - FAQ

Common questions about deploying and using CAPT.

How is this different from using a general-purpose speech recognizer?

A recognizer is built to recover the intended message. That makes it actively unsuitable for assessment, because it will repair what it hears into what it assumes you meant: a mispronounced word is transcribed as the word, and the error you were trying to measure disappears. The better the language model, the more thoroughly it hides exactly the signal you need.

CAPT is anchored to a reference you supply and never substitutes expectation for observation. It also gives you things a transcript cannot:

  • Per-phoneme verdicts and scores, not just a word-level transcript.
  • Confidence taken from a full distribution over phonemes, not a single guess.
  • Timestamps for each recognized phoneme.
  • Deterministic, inspectable scoring you can tune, with no risk of a generative model inventing plausible text.
  • Operation entirely on your own hardware, on CPU.

What accuracy should I expect?

It depends on the task, the audio and the speakers, so we do not publish a single figure that would mislead you. What we would recommend instead is measuring it on your own data, against the decisions you actually care about.

If you are automating a judgement a person currently makes, the number that matters is agreement with that person on your items, not word error rate, and not any generic benchmark. See calibrating against human judgement.

Does audio leave my infrastructure?

No. CAPT runs on your own hardware: on-premise, in your private cloud, or embedded on a device. There is no call home, and no dependency on an external service at request time.

Audio and results are not stored unless you explicitly enable it. Storage is off by default; to turn it on, set a storage backend and path in capt-server.cfg.toml:

[storage]
Type = "localfs"
BasePath = "/audio"

With this enabled, each session is written to local disk as two files, the audio and the evaluation result, organized by UTC date. Everything stays on the machine you run.

Note that metadata.custom_id and metadata.custom_metadata may be written to logs and to stored results, so they should carry opaque identifiers rather than personal information.

What hardware does it need?

CAPT is CPU-only; no GPU is required. It runs on x86_64 and Arm64 / aarch64, and a statically linked build is available for minimal or embedded images.

Sizing depends on the model and on how many concurrent evaluations you need. Contact us for guidance against your target hardware.

What languages are supported?

Each model covers one language, and the model’s language is fixed at build time. US English models are available today, and models for other languages can be built. Contact us to discuss a specific language.

What audio should I send?

Use uncompressed or losslessly compressed audio (WAV or FLAC) recorded at the model’s native sample rate, which ListModels reports as attributes.sample_rate.

Audio sampled below what the model expects yields a non-fatal warning and reduced accuracy. Upsampling before sending does not help; the detail is already gone. Lossy formats such as MP3 are accepted but will cost you some accuracy.

Can more than one person be speaking in the recording?

CAPT scores the audio against the reference text; it does not separate speakers. If someone other than the intended speaker is audible, an assistant reading a prompt for instance, that speech is part of the audio being evaluated and can influence the result, usually appearing as insertions.

Where recordings may contain more than one voice, control it at capture time: record only while the intended speaker is expected to be talking, rather than across the whole interaction. If that is not possible for your setup, talk to us about the options.

Can I add my own words and pronunciations?

Yes, and this is a normal thing to do. Add them to lexicon_addenda.tsv in the model directory. You can add words the lexicon does not have, and override the pronunciations of words it does. Words in neither the lexicon nor the addenda are handled automatically by a grapheme-to-phoneme model, so invented words and non-words work without any setup.

See Tuning and Customization.

Can I make scoring stricter or more lenient?

Yes, globally or per phoneme, by editing configuration rather than retraining. DefaultCutoffScore moves the bar for everything; cutoffs.yaml sets it sound by sound, and can additionally name specific confusions that should be tolerated. See Cutoffs and acceptable confusions.

Do I have to use streaming?

No. Evaluate takes the whole audio in one request and returns one result, which is the simplest option for scoring a file. Use StreamingEvaluate when you want results while the speaker is still talking, or for audio longer than about a minute.

Are partial results guaranteed?

No. Servers are not required to produce them, and clients should not depend on them. Drive your logic from the final result, the one with is_partial unset or false, and treat partials as an enhancement for live feedback.

What does a score of 0.5 mean?

It means “exactly borderline” for that phoneme. Scores are not raw probabilities: each phoneme has a cutoff, and the confidence-to-score mapping is built so that the cutoff lands on 0.5. That is what makes 0.5 a meaningful threshold and lets different phonemes be held to different standards.

Overall utterance scores are a summary of many such phoneme scores, so an overall 0.5 does not mean “half correct”. Look at the per-token verdicts before thresholding. See Interpreting Results.

4 - Voice User Interfaces

Demos

4.1 - VoiceGen

Low latency, on-prem / on-cloud solutions for highly natural streaming text to speech synthesis.

VoiceGen synthesizes speech from text. The API has a single synthesis call, StreamingSynthesize: you send the text and a SynthesisConfig naming the model, the speaker and the audio format you want, and the server streams audio back in chunks as it is produced — so playback can start before the whole utterance is synthesized.

ListModels reports which voices a server has loaded, including the speakers available in each model and the phone set it accepts, which is what you need if you plan to send pronunciations rather than plain text.

Like the other engines, VoiceGen runs on your own hardware, on prem, in your private cloud, or embedded.

4.1.1 - Getting Started

How to get a VoiceGen Server running on your system

Using Cobalt VoiceGen

  • A typical VoiceGen release, provided as a compressed archive, will contain a linux binary (voicegen-server) for the required native CPU architecture, appropriate Dockerfile and models.

  • Cobalt VoiceGen runs either locally on linux or using Docker.

  • Cobalt VoiceGen will serve the GRPC API on port 2727. A web demo will be enabled on port 8080.

  • To quickly try out VoiceGen, first start the server as shown below and open the web demo at http://localhost:8080 in your browser to input text and play / download synthesized audio. You can also use the SDK in your preferred language to use VoiceGen from the command line or within your application.

Running VoiceGen Server Locally on Linux

./voicegen-server
  • By default, the binary assumes the presence of a configuration file, located in the same directory, named: voicegen-server.cfg.toml. A different config file may be specified using the --config argument.

Running VoiceGen Server as a Docker Container

To build and run the Docker image for VoiceGen, run:

docker build -t cobalt-voicegen .
docker run -p 2727:2727 -p 8080:8080 cobalt-voicegen

How to Get a Copy of the VoiceGen Server and Models

Contact us for getting a release best suited to your requirements.

The release you will receive is a compressed archive (tar.bz2) and is generally structured accordingly:

release.tar.bz2
├── COPYING
├── README.md
├── voicegen-server
├── voicegen-server.cfg.toml
├── Dockerfile
├── models
│   └── en_US-multispeaker-22050hz
│
└── cobalt.license.key [ provided separately, needs to be copied over ]
  • The README.md file contains information about this release and instructions for how to start the server on your system.

  • The voicegen-server is the server program which is configured using the voicegen-server.cfg.toml file.

  • The Dockerfile can be used to create a container that will let you run VoiceGen server on non-linux systems such as MacOS and Windows.

  • The models directory contains the speech synthesis models. The content of these directory will depend on the models you are provided.

System Requirements

Cobalt VoiceGen runs on Linux. You can run it directly as a linux application.

You can evaluate the product on Windows or Linux using Docker Desktop but we would not recommend this setup for use in a production environment.

A Cobalt VoiceGen release typically includes a single model together with binaries and config files. VoiceGen models may take up to 250MB of disk space, and need a minimum of 2GB RAM when evaluating locally. For production workloads, we recommend configuring containerized applications with each instance allocated with 4 CPUs and 4GB RAM.

Cobalt VoiceGen runs on x86_64 CPUs. We also support Arm64 CPUs, including processors such as the Graviton (AWS c7g EC2 instances). VoiceGen is significantly more cost effective to run on C7g instances compared to similarly sized Intel or AMD processors, and we can provide you an Arm64 release on request.

To integrate Cobalt VoiceGen into your application, please follow the next steps to install or generate the SDK in a language of your choice.

4.1.2 - Generating SDKs

Gives instructions about how to generate an SDK for your project from the proto API definition.
  • APIs for all Cobalt’s services are defined as a protocol buffer specification or simply a proto file and be found in the cobaltspeech/proto github repository.

  • The proto file allows a developer to auto-generate client SDKs for a number of different programming languages. Step by step instructions for generating your own SDK can be found below.

  • We provide pre-generated SDKs for a couple of languages. You can choose to use these instead of generating your own. These are listed here along with instructions on how to install / import them into your projects.

Pre-generated SDKs

Golang

import voicegenpb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicegen/v1"
  • An example client using the above repo can be found here.

Python

  • Pre-generated SDK files for Python can be found in the cobaltspeech/py-genproto repo

  • The Python SDK depends on Python >= 3.5. You may use pip to perform a system-wide install, or use virtualenv for a local install. To use it in your Python project, install it:

pip install --upgrade pip
pip install "git+https://github.com/cobaltspeech/py-genproto"

Generating SDKs

Step 1. Installing buf

  • To work with proto files, we recommend using buf, a user-friendly command line tool that can be configured generate documentation, schemas and SDK code for different languages.
# Latest version as of March 14th, 2023.

COBALT="${HOME}/cobalt"
  mkdir -p "${COBALT}/bin"

VERSION="1.15.1"
URL="https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)"
  curl -L ${URL} -o "${COBALT}/bin/buf"

# Give executable permissions and adding to $PATH.

chmod +x "${COBALT}/bin/buf"
  export PATH="${PATH}:${COBALT}/bin"
brew install bufbuild/buf/buf

Step 2. Getting proto files

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Change this to where you want to clone the repo to.
PROTO_REPO="${COBALT}/git/proto"

git clone https://github.com/cobaltspeech/proto "${PROTO_REPO}"

Step 3. Generating code

  • The cobaltspeech/proto repo provides a buf.gen.yaml config file to get you started with a couple of languages.

  • Other plugins can be added to the buf.gen.yaml file to generate SDK code for more languages.

  • To generate the SDKs, simply run the following (assuming the buf binary is in your $PATH)

cd "${PROTO_REPO}"

# Removing any previously generated files.
rm -rf ./gen

# Generating code for all proto files inside the `proto` directory.
buf generate proto
  • You should now have a folder called gen inside ${PROTO_REPO} that contains the generated code. The latest version of the VoiceGen API is v1. You can import / include / copy the generated files into your projects as per the conventions of different languages.
gen
├── ... other languages ...
└── py
  └── cobaltspeech
    ├── ... other services ...
    └── voicegen
      └── v1
        ├── voicegen_pb2_grpc.py
        ├── voicegen_pb2.py
        └── voicegen_pb2.pyi
gen
├── ... other languages ...
└── go
   ├── cobaltspeech
   │ ├── ...
   │   └── voicegen
   │      └── v1
   │        ├── voicegen_grpc.pb.go
   │        └── voicegen.pb.go
   └── gw
     └── cobaltspeech
       ├── ...
       └── voicegen
         └── v1
            └── voicegen.pb.gw.go

Step 4. Installing gPRC and protobuf

  • A couple of gRPC and protobuf dependencies are required along with the code generated above. The method of installing them depends on the programming language being used.
  • These dependencies and the most common way of installing/ / including them are listed below for some chosen languages.
# It is encouraged to this inside a python virtual environment

# to avoid creating version conflicts for other scripts that may

# be using these libraries.

pip install --upgrade protobuf
pip install --upgrade grpcio
pip install --upgrade google-api-python-client
go get google.golang.org/protobuf
go get google.golang.org/grpc
go get google.golang.org/genproto
# More details on grpc installation can be found at:

# https://grpc.io/docs/languages/cpp/quickstart/

COBALT="${HOME}/cobalt"
mkdir -p "${COBALT}/git"

# Latest version as of 14th March, 2023.

VERSION="v1.52.0"
GRPC_REPO="${COBALT}/git/grpc-${VERSION}"

git clone \
 --recurse-submodules --depth 1 --shallow-submodules \
 -b "${VERSION}" \
 https://github.com/grpc/grpc ${GRPC_REPO}

cd "${GRPC_REPO}"
mkdir -p cmake/build

# Change this to where you want to install libprotobuf and libgrpc.

# It is encouraged to install gRPC locally as there is no easy way to

# uninstall gRPC after you’ve installed it globally.

INSTALL_DIR="${COBALT}"

cd cmake/build
cmake \
 -DgRPC_INSTALL=ON \
 -DgRPC_BUILD_TESTS=OFF \
 -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR} \
 ../..

make -j
make install

4.1.3 - Connecting to the Server

Describes how to connect to a running Cobalt VoiceGen server instance.
  • Once you have your VoiceGen server up and running, and have installed or generated the SDK for your project, you can connect to a running instance of VoiceGen server, by “dialing” a gRPC connection.

  • First, you need to know the address where the server is running: e.g. host:grpc_port. By default, this is localhost:2727 and should be logged to the terminal when you first start VoiceGen server as grpcAddr:

2023/08/14 10:49:38 info  {"license":"Copyright © 2015--present. Cobalt Speech and Language, Inc.  For additional details, including information about open source components used in this software, please see the COPYING file bundled with this program."}
2023/08/14 10:49:38 info  {"msg":"reading config file","path":"configs/voicegen-server.config.toml"}
2023/08/14 10:49:38 info  {"msg":"server initializing"}
2023/08/14 10:49:41 info  {"msg":"server started","grpcAddr":"[::]:2727","httpApiAddr":"[::]:8080","httpOpsAddr":"[::]:8081"}

Default Connection

The following code snippet connects to the server and queries its version. It connects to the server using an “insecure” gRPC channel. This would be the case if you have just started up a local instance of VoiceGen server without TLS enabled.

import grpc
import cobaltspeech.voicegen.v1.voicegen_pb2_grpc as stub
import cobaltspeech.voicegen.v1.voicegen_pb2 as voicegen

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceGenServiceStub(channel)

# Get server version.
versionResp = client.Version(voicegen.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"fmt"
	"os"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicegenpb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicegen/v1"
)

func main() {
	const (
		serverAddress  = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicegenpb.NewVoiceGenServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicegenpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Connect with TLS

  • In our recommended setup for deployment, TLS is enabled in the gRPC connection, and when connecting to the server, clients validate the server’s SSL certificate to make sure they are talking to the right party. This is similar to how “https” connections work in web browsers.

  • The following snippets show how to connect to a VoiceGen Server that has TLS enabled. They use the cobalt’s self-hosted demo server at demo.cobaltspeech.com:2727, but you obviously use your own server instance.

import grpc
import cobaltspeech.voicegen.v1.voicegen_pb2_grpc as stub
import cobaltspeech.voicegen.v1.voicegen_pb2 as voicegen

serverAddress = "demo.cobaltspeech.com:2727"

# Setup a gRPC connection with TLS. You can optionally provide your own
# root certificates and private key to grpc.ssl_channel_credentials()
# for mutually authenticated TLS.
creds = grpc.ssl_channel_credentials()
channel = grpc.secure_channel(serverAddress, creds)
client = stub.VoiceGenServiceStub(channel)

# Get server version.
versionResp = client.Version(voicegen.VersionRequest())
print(versionResp)
package main

import (
	"context"
	"crypto/tls"
	"fmt"
	"os"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"

	voicegenpb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicegen/v1"
)

func main() {
	const (
		serverAddress  = "demo.cobaltspeech.com:2727"
		connectTimeout = 10 * time.Second
	)

	// Setup a gRPC connection with TLS. You can optionally provide your own
	// root certificates and private key through tls.Config for mutually
	// authenticated TLS.
	tlsCfg := tls.Config{}
	creds := credentials.NewTLS(&tlsCfg)

	ctx, cancel := context.WithTimeout(context.Background(), connectTimeout)
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(creds),
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicegenpb.NewVoiceGenServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicegenpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)
}

Client Authentication

  • In some setups, it may be desired that the server should also validate clients connecting to it and only respond to the ones it can verify. If your VoiceGen server is configured to do client authentication, you will need to present the appropriate certificate and key when connecting to it.

  • Please note that in the client-authentication mode, the client will still also verify the server’s certificate, and therefore this setup uses mutually authenticated TLS.

  • The following snippets show how to present client certificates when setting up the credentials. These could then be used in the same way as the examples above to connect to a TLS enabled server.

creds = grpc.ssl_channel_credentials(
  root_certificates=root_certificates,  # PEM certificate as byte string
  private_key=private_key,              # PEM client key as byte string 
  certificate_chain=certificate_chain,  # PEM client certificate as byte string
)
package main

import (
	// ...

	"crypto/tls"
	"crypto/x509"
	"fmt"
	"os"

	// ..
)

func main() {
	// ...

	// Root PEM certificate for validating self-signed server certificate
	var rootCert []byte

	// Client PEM certificate and private key.
	var certPem, keyPem []byte

	caCertPool := x509.NewCertPool()
	if ok := caCertPool.AppendCertsFromPEM(rootCert); !ok {
		fmt.Printf("unable to use given caCert\n")
		os.Exit(1)
	}

	clientCert, err := tls.X509KeyPair(certPem, keyPem)
	if err != nil {
		fmt.Printf("unable to use given client certificate and key: %v\n", err)
		os.Exit(1)
	}

	tlsCfg := tls.Config{
		RootCAs:      caCertPool,
		Certificates: []tls.Certificate{clientCert},
	}

	creds := credentials.NewTLS(&tlsCfg)

	// ...
}

4.1.4 - Streaming Synthesis

Describes how to submit text VoiceGen server for streaming synthesis.
  • The following example shows how to synthesize streaming audio from text using VoiceGen’s StreamingSynthesize request. The audio can be played back as it is being streamed as well as being saved to a file or buffer.

Synthesizing streaming audio and writing to a file

  • We support streaming several headered file formats including WAV, MP3, FLAC etc. as well streaming raw audio samples. For more details, please see the protocol buffer specification here.

  • The examples below show how to submit a chunk of text and receive streaming audio which is written to a file. We will query the server for available models and use the first model for synthesis.

import grpc
import cobaltspeech.voicegen.v1.voicegen_pb2_grpc as stub
import cobaltspeech.voicegen.v1.voicegen_pb2 as voicegen

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceGenServiceStub(channel)

# Get server version.
versionResp = client.Version(voicegen.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicegen.ListModelsRequest())

# A model may be a single-speaker model or a multi-speaker model.
# The speakers available for a model will be printed in the model
# attributes below.
print("Models:")
for model in modelResp.models:
    print(model)

# Going with the first model in this example. Also using the first
# speaker available in the model (in case of single-speaker models,
# it is the *only* speaker).
model = modelResp.models[0]
spk = model.attributes.speakers[0]

# Set the synthesis config.
# 
# - We could set speaker_id to None to let the server use the default
#   speaker configured on the server side.
#
# - We are specifying the output audio format to be WAV with 16 bit signed
#   samples, at the model's native sampling rate.
cfg = voicegen.SynthesisConfig(
    model_id=model.id,
    speaker_id=spk.id,
    audio_format=voicegen.AudioFormat(
      codec=voicegen.AUDIO_CODEC_WAV,
	  sample_rate=model.attributes.native_audio_format.sample_rate,
      encoding=voicegen.AUDIO_ENCODING_SIGNED,
      bit_depth=16,
	  channels=1,
      byte_order=voicegen.BYTE_ORDER_LITTLE_ENDIAN,
    ),
)

# Specifying text to synthesize, which could be a single line or multiple paragraphs.
# VoiceGen breaks up the text based on its sentence segmentation algorithm as well as
# any line breaks specified in the input text. We intentionally put line breaks here
# to make it look a bit nicer in the code, which are replaced with spaces.
text = voicegen.SynthesisText(text='''
The world's first 3D printed rocket launched successfully on Wednesday, marking
a step forward for the California company behind the innovative spacecraft,
though it failed to reach orbit.

The successful launch came on the third attempt. It had originally been
scheduled to launch on March 8 but was postponed at the last minute because of
propellant temperature issues. A second attempt on March 11 was scrubbed because of
fuel pressure problems.

Had Terran 1 reached low Earth orbit, it would have been the first privately
funded vehicle using methane fuel to do so on its first try, according to
Relativity.
'''.replace("\n", " "))

# Submitting request to the server and writing streamed audio chunks to file.
print("Synthesizing ...")
with open("output.wav", 'wb') as f:
    for resp in client.StreamingSynthesize(voicegen.StreamingSynthesizeRequest(config=cfg, text=text)):
	    f.write(resp.audio.data)
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"strings"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicegenpb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicegen/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicegenpb.NewVoiceGenServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicegenpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicegenpb.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	// A model may be a single-speaker model or a multi-speaker model.
	// The speakers available for a model will be printed in the model
	// attributes below.
	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Going with the first model in this example. Also using the first
	// speaker available in the model (in case of single-speaker models,
	// it is the *only* speaker).
	model := modelResp.Models[0]
	spk := model.Attributes.Speakers[0]

	// Set the synthesis config.
	//
	//   - We could set speaker_id to None to let the server use the default
	//     speaker configured on the server side.
	//
	//   - We are specifying the output audio format to be WAV with 16 bit signed
	//     samples, at the model's native sampling rate.
	cfg := &voicegenpb.SynthesisConfig{
		ModelId:   model.Id,
		SpeakerId: spk.Id,
		AudioFormat: &voicegenpb.AudioFormat{
			Codec:      voicegenpb.AudioCodec_AUDIO_CODEC_WAV,
			SampleRate: model.Attributes.NativeAudioFormat.SampleRate,
			Encoding:   voicegenpb.AudioEncoding_AUDIO_ENCODING_SIGNED,
			BitDepth:   16,
			Channels:   1,
			ByteOrder:  voicegenpb.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
		},
	}

	// Specifying text to synthesize, which could be a single line or multiple paragraphs.
	// VoiceGen breaks up the text based on its sentence segmentation algorithm as well as
	// any line breaks specified in the input text. We intentionally put line breaks here
	// to make it look a bit nicer in the code, which are replaced with spaces.
	text := &voicegenpb.SynthesisText{Text: strings.ReplaceAll(`
The world's first 3D printed rocket launched successfully on Wednesday, marking
a step forward for the California company behind the innovative spacecraft,
though it failed to reach orbit.

The successful launch came on the third attempt. It had originally been
scheduled to launch on March 8 but was postponed at the last minute because of
propellant temperature issues. A second attempt on March 11 was scrubbed because of
fuel pressure problems.

Had Terran 1 reached low Earth orbit, it would have been the first privately
funded vehicle using methane fuel to do so on its first try, according to
Relativity.
`, "\n", " ")}

	// Submitting request to the server and writing streamed audio chunks to file.
	fmt.Println("Synthesizing ...")
	stream, err := client.StreamingSynthesize(context.Background(), &voicegenpb.StreamingSynthesizeRequest{Config: cfg, Text: text})
	if err != nil {
		fmt.Printf("failed to start synthesis stream: %v\n", err)
		os.Exit(1)
	}

	// Opening output audio file.
	outF, err := os.Create("output.wav")
	if err != nil {
		fmt.Printf("failed to open output audio file: %v\n", err)
		os.Exit(1)
	}

	defer outF.Close()

	// Receiving audio and writing to file.
	for {
		resp, err := stream.Recv()
		if errors.Is(io.EOF, err) {
			return
		}

		if err != nil {
			fmt.Printf("error encountered while synthesizing: %v\n", err)
			os.Exit(1)
		}

		audio := resp.GetAudio()
		if audio == nil {
			fmt.Printf("error encountered while synthesizing: server returned nil audio")
			os.Exit(1)
		}

		outF.Write(audio.Data)
	}
}

Synthesizing streaming audio with live playback

  • The synthesized audio stream can be played back live instead of saving it to a file by writing the data to an appropriate interface that can do the playback; typically this requires interaction with system libraries. Another option is to pipe the audio out to an external command line tool like sox.

  • The examples below use the latter approach by using the play command provided with sox to play the synthesized audio stream live.

import subprocess
import grpc
import cobaltspeech.voicegen.v1.voicegen_pb2_grpc as stub
import cobaltspeech.voicegen.v1.voicegen_pb2 as voicegen

serverAddress = "localhost:2727"

# Using a channel without TLS enabled.
channel = grpc.insecure_channel(serverAddress)
client = stub.VoiceGenServiceStub(channel)

# Get server version.
versionResp = client.Version(voicegen.VersionRequest())
print(versionResp)

# Get list of models on the server.
modelResp = client.ListModels(voicegen.ListModelsRequest())

# A model may be a single-speaker model or a multi-speaker model.
# The speakers available for a model will be printed in the model
# attributes below.
print("Models:")
for model in modelResp.models:
    print(model)

# Going with the first model in this example. Also using the first
# speaker available in the model (in case of single-speaker models,
# it is the *only* speaker).
model = modelResp.models[0]
spk = model.attributes.speakers[0]

# Set the synthesis config.
# 
# - We could set speaker_id to None to let the server use the default
#   speaker configured on the server side.
#
# - We are specifying the output audio format to be WAV with 16 bit signed
#   samples, at the model's native sampling rate.
cfg = voicegen.SynthesisConfig(
    model_id=model.id,
    speaker_id=spk.id,
    audio_format=voicegen.AudioFormat(
      codec=voicegen.AUDIO_CODEC_WAV,
	  sample_rate=model.attributes.native_audio_format.sample_rate,
      encoding=voicegen.AUDIO_ENCODING_SIGNED,
      bit_depth=16,
	  channels=1,
      byte_order=voicegen.BYTE_ORDER_LITTLE_ENDIAN,
    ),
)

# Specifying text to synthesize, which could be a single line or multiple paragraphs.
# VoiceGen breaks up the text based on its sentence segmentation algorithm as well as
# any line breaks specified in the input text. We intentionally put line breaks here
# to make it look a bit nicer in the code, which are replaced with spaces.
text = voicegen.SynthesisText(text='''
The world's first 3D printed rocket launched successfully on Wednesday, marking
a step forward for the California company behind the innovative spacecraft,
though it failed to reach orbit.

The successful launch came on the third attempt. It had originally been
scheduled to launch on March 8 but was postponed at the last minute because of
propellant temperature issues. A second attempt on March 11 was scrubbed because of
fuel pressure problems.

Had Terran 1 reached low Earth orbit, it would have been the first privately
funded vehicle using methane fuel to do so on its first try, according to
Relativity.
'''.replace("\n", " "))

# Open playback stream using sox's play command as subprocess.
cmd = f"play -t wav -"
play = subprocess.Popen(cmd.split(), stdin=subprocess.PIPE)
out = play.stdin

# Submitting request to the server and writing streamed audio chunks to playback stream.
print("Synthesizing ...")
for resp in client.StreamingSynthesize(voicegen.StreamingSynthesizeRequest(config=cfg, text=text)):
    out.write(resp.audio.data)

out.close()
play.wait()
play.kill()
package main

import (
	"context"
	"errors"
	"fmt"
	"io"
	"os"
	"os/exec"
	"strings"

	"golang.org/x/sync/errgroup"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	voicegenpb "github.com/cobaltspeech/go-genproto/cobaltspeech/voicegen/v1"
)

func main() {
	const (
		serverAddress = "localhost:2727"
	)

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	opts := []grpc.DialOption{
		grpc.WithTransportCredentials(insecure.NewCredentials()), // Using a channel without TLS enabled.
		grpc.WithBlock(),
		grpc.WithReturnConnectionError(),
		grpc.FailOnNonTempDialError(true),
	}

	conn, err := grpc.DialContext(ctx, serverAddress, opts...)
	if err != nil {
		fmt.Printf("failed to dial gRPC connection: %v\n", err)
		os.Exit(1)
	}

	client := voicegenpb.NewVoiceGenServiceClient(conn)

	// Get server version.
	versionResp, err := client.Version(ctx, &voicegenpb.VersionRequest{})
	if err != nil {
		fmt.Printf("failed to get server version: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("%v\n", versionResp)

	// Get list model of models on the server.
	modelResp, err := client.ListModels(ctx, &voicegenpb.ListModelsRequest{})
	if err != nil {
		fmt.Printf("failed to get model list: %v\n", err)
		os.Exit(1)
	}

	// A model may be a single-speaker model or a multi-speaker model.
	// The speakers available for a model will be printed in the model
	// attributes below.
	fmt.Println("Models:")
	for _, m := range modelResp.Models {
		fmt.Println(m)
	}
	fmt.Println()

	// Going with the first model in this example. Also using the first
	// speaker available in the model (in case of single-speaker models,
	// it is the *only* speaker).
	model := modelResp.Models[0]
	spk := model.Attributes.Speakers[0]

	// Set the synthesis config.
	//
	//   - We could set speaker_id to None to let the server use the default
	//     speaker configured on the server side.
	//
	//   - We are specifying the output audio format to be WAV with 16 bit signed
	//     samples, at the model's native sampling rate.
	cfg := &voicegenpb.SynthesisConfig{
		ModelId:   model.Id,
		SpeakerId: spk.Id,
		AudioFormat: &voicegenpb.AudioFormat{
			Codec:      voicegenpb.AudioCodec_AUDIO_CODEC_WAV,
			SampleRate: model.Attributes.NativeAudioFormat.SampleRate,
			Encoding:   voicegenpb.AudioEncoding_AUDIO_ENCODING_SIGNED,
			BitDepth:   16,
			Channels:   1,
			ByteOrder:  voicegenpb.ByteOrder_BYTE_ORDER_LITTLE_ENDIAN,
		},
	}

	// Specifying text to synthesize, which could be a single line or multiple paragraphs.
	// VoiceGen breaks up the text based on its sentence segmentation algorithm as well as
	// any line breaks specified in the input text. We intentionally put line breaks here
	// to make it look a bit nicer in the code, which are replaced with spaces.
	text := &voicegenpb.SynthesisText{Text: strings.ReplaceAll(`
The world's first 3D printed rocket launched successfully on Wednesday, marking
a step forward for the California company behind the innovative spacecraft,
though it failed to reach orbit.

The successful launch came on the third attempt. It had originally been
scheduled to launch on March 8 but was postponed at the last minute because of
propellant temperature issues. A second attempt on March 11 was scrubbed because of
fuel pressure problems.

Had Terran 1 reached low Earth orbit, it would have been the first privately
funded vehicle using methane fuel to do so on its first try, according to
Relativity.
`, "\n", " ")}

	// Starting routines to receive audio from server and write to playback stream;
	// using an errgroup.Group that returns if either one encounters an error.
	eg, ctx := errgroup.WithContext(context.Background())

	// Submitting request to the server and writing streamed audio chunks to file.
	fmt.Println("Synthesizing ...")
	stream, err := client.StreamingSynthesize(ctx, &voicegenpb.StreamingSynthesizeRequest{Config: cfg, Text: text})
	if err != nil {
		fmt.Printf("failed to start synthesis stream: %v\n", err)
		os.Exit(1)
	}

	// Open playback stream using sox's play command as a subprocess.
	cmd := exec.CommandContext(ctx, "play", "-t", "wav", "-")
	cmd.Stderr = os.Stderr

	outW, err := cmd.StdinPipe()
	if err != nil {
		fmt.Printf("failed to open playback stream: %v\n", err)
		os.Exit(1)
	}

	eg.Go(func() error {
		if err := cmd.Run(); err != nil {
			return fmt.Errorf("error encountered in audio playback: %w", err)
		}

		return nil
	})

	eg.Go(func() error {
		defer outW.Close()

		// Receiving audio and writing to playback stream.
		for {
			resp, err := stream.Recv()
			if errors.Is(io.EOF, err) {
				return nil
			}

			if err != nil {
				return fmt.Errorf("error encountered while synthesizing: %w", err)
			}

			audio := resp.GetAudio()
			if audio == nil {
				return fmt.Errorf("error encountered while synthesizing: server returned nil audio")
			}

			outW.Write(audio.Data)
		}
	})

	if err := eg.Wait(); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
}

4.1.5 - API Reference

Detailed reference for API requests and types.

The API is defined as a protobuf spec, so native bindings can be generated in any language with gRPC support. We recommend using buf to generate the bindings.

This section of the documentation is auto-generated from the protobuf spec. The service contains the methods that can be called, and the “messages” are the data structures (objects, classes or structs in the generated code, depending on the language) passed to and from the methods.

VoiceGenService

Service that implements the Cobalt VoiceGen API.

Version

Version(VersionRequest) VersionResponse

Returns version information from the server.

ListModels

ListModels(ListModelsRequest) ListModelsResponse

ListModels returns information about the models the server can access.

StreamingSynthesize

StreamingSynthesize(StreamingSynthesizeRequest) StreamingSynthesizeResponse

Performs text to speech synthesis and stream synthesized audio. This method is only available via GRPC and not via HTTP+JSON. However, a web browser may use websockets to use this service.

Messages

  • If two or more fields in a message are labeled oneof, then each method call using that message must have exactly one of the fields populated
  • If a field is labeled repeated, then the generated code will accept an array (or struct, or list depending on the language).

AudioFormat

Details of audio in format

Fields

  • sample_rate (uint32 ) Sampling rate in Hz.

  • channels (uint32 ) Number of channels present in the audio. E.g.: 1 (mono), 2 (stereo), etc.

  • bit_depth (uint32 ) Bit depth of each sample (e.g. 8, 16, 24, 32, etc.).

  • codec (AudioCodec ) Codec of the samples.

  • encoding (AudioEncoding ) Encoding of the samples.

  • byte_order (ByteOrder ) Byte order of the samples. This field must be set to a value other than BYTE_ORDER_UNSPECIFIED when the bit_depth is greater than 8.

ListModelsRequest

The top-level message sent by the client for the ListModels method.

ListModelsResponse

The message returned to the client by the ListModels method.

Fields

  • models (ModelInfo repeated) List of models available for use.

ModelAttributes

Attributes of a VoiceGen Model

Fields

  • language (string ) Language of the model.

  • phone_set (PhoneSet ) The set of phonemes this model uses to represent how words should be pronounced.

  • native_audio_format (AudioFormat ) Native audio format of the model. This will be use as default value if audio format in SynthesisConfig is not specify.

  • supported_features (ModelFeatures ) Supported model features.

  • speakers (SpeakerInfo repeated) List of speaker available for use in this model.

ModelFeatures

Fields

  • speech_rate (bool ) This is set to true if the model can be configured to synthesize audio at different talking speeds.

  • variation_scale (bool ) This is set to true if the model can be configured to synthesize audio for a given text input differently than usual by varying stresses, and emphasis on different parts of the audio. This feature is useful for making the audio sound slightly different each time to avoid making it feel monotonous.

ModelInfo

Description of a Cobalt VoiceGen Model

Fields

  • id (string ) Unique identifier of the model. This identifier is used to choose the model that should be used for synthesis, and is specified in the SynthesisConfig message.

  • name (string ) Model name. This is a concise name describing the model, and may be presented to the end-user, for example, to help choose which model to use for their synthesis task.

  • attributes (ModelAttributes ) Model attributes.

SpeakerAttributes

Attributes of a speaker

Fields

  • language (string ) Language of the speaker. This can be different from model language. E.g. an english model with different accents: en-US, en-GB, en-IN etc.

SpeakerInfo

Description of a speaker

Fields

  • id (string ) Unique identifier of the speaker. This identifier is used to choose the speaker that should be used for synthesis, and is specified in the SynthesisConfig message.

  • name (string ) Speaker name. This is a concise name describing the speaker, and may be presented to the end-user, for example, to help choose which speaker to use for their synthesis task.

  • description (string ) Speaker description. This is may be presented to the end-user, for example, to help choose which speaker to use for their synthesis task.

  • attributes (SpeakerAttributes ) Speaker attributes.

StreamingSynthesizeRequest

The top-level messages sent by the client for the StreamingSynthesize method.

Fields

StreamingSynthesizeResponse

The top-level message sent by the server for the StreamingSynthesize method. In this streaming call, multiple StreamingSynthesizeResponse messages contain SynthesizedAudio.

Fields

SynthesisConfig

Configuration for setting up a Synthesizer

Fields

  • model_id (string ) Unique identifier of the model to use, as obtained from a ModelInfo message.

  • speaker_id (string ) Unique identifier of the speaker to use, as obtained from a SpeakerInfo message.

  • audio_format (AudioFormat ) Format of the audio to be sent for synthesis. If no value specify, default value of native audio format of the specified model will be used. Native audio format can be obtained from ModelAttributes message.

  • speech_rate (float ) The speech rate for synthesized audio. If unset, then the default speech rate of a given model is used. Otherwise a value > 0 should be used, with higher values resulting in faster speech. This field only has an effect on the synthesized audio if the model supports it, which can be ascertained from the ModelAttributes.supported_features.

  • variation_scale (float ) A scale with values > 0, to determine how much to randomly vary the synthesized audio by altering stresses and emphasis on different parts of the audio. Higher values correspond to greater variation. This field only has an affect on the synthesized audio if the model supports it, which can be ascertained from the ModelAttributes.supported_features.

SynthesisText

Text input to be sent to the synthesizer

Fields

SynthesizedAudio

Synthesize audio from the synthesizer

Fields

VersionRequest

The top-level message sent by the client for the Version method.

VersionResponse

The top-level message sent by the server for the Version method.

Fields

  • version (string ) Version of the server handling these requests.

Enums

AudioCodec

The encoding of the audio data to be sent for synthesis.

NameNumberDescription
AUDIO_CODEC_UNSPECIFIED0AUDIO_CODEC_UNSPECIFIED is the default value of this type.
AUDIO_CODEC_RAW2Raw data without any headers
AUDIO_CODEC_WAV1WAV with RIFF headers

AudioEncoding

The encoding of the audio data to be sent for synthesis.

NameNumberDescription
AUDIO_ENCODING_UNSPECIFIED0AUDIO_ENCODING_UNSPECIFIED is the default value of this type and will result in an error.
AUDIO_ENCODING_SIGNED1PCM signed-integer
AUDIO_ENCODING_UNSIGNED2PCM unsigned-integer
AUDIO_ENCODING_IEEE_FLOAT3PCM IEEE-Float
AUDIO_ENCODING_ULAW4G.711 mu-law
AUDIO_ENCODING_ALAW5G.711 a-law

ByteOrder

Byte order of multi-byte data

NameNumberDescription
BYTE_ORDER_UNSPECIFIED0BYTE_ORDER_UNSPECIFIED is the default value of this type.
BYTE_ORDER_LITTLE_ENDIAN1Little Endian byte order
BYTE_ORDER_BIG_ENDIAN2Big Endian byte order

PhoneSet

PhoneSet is a set of phonemes for words pronunciation.

NameNumberDescription
PHONE_SET_UNSPECIFIED0PHONE_SET_UNSPECIFIED is the default value of this type.
PHONE_SET_IPA1IPA phoneme set
PHONE_SET_XSAMPA2X-SAMPA phoneme set
PHONE_SET_ARPABET3ARPAbet phoneme set

Scalar Value Types

.proto TypeC++ TypeC# TypeGo TypeJava TypePHP TypePython TypeRuby Type

double
doubledoublefloat64doublefloatfloatFloat

float
floatfloatfloat32floatfloatfloatFloat

int32
int32intint32intintegerintBignum or Fixnum (as required)

int64
int64longint64longinteger/stringint/longBignum

uint32
uint32uintuint32intintegerint/longBignum or Fixnum (as required)

uint64
uint64ulonguint64longinteger/stringint/longBignum or Fixnum (as required)

sint32
int32intint32intintegerintBignum or Fixnum (as required)

sint64
int64longint64longinteger/stringint/longBignum

fixed32
uint32uintuint32intintegerintBignum or Fixnum (as required)

fixed64
uint64ulonguint64longinteger/stringint/longBignum

sfixed32
int32intint32intintegerintBignum or Fixnum (as required)

sfixed64
int64longint64longinteger/stringint/longBignum

bool
boolboolboolbooleanbooleanbooleanTrueClass/FalseClass

string
stringstringstringStringstringstr/unicodeString (UTF-8)

bytes
stringByteString[]byteByteStringstringstrString (ASCII-8BIT)