Running Whisper inference using OpenVINO

This PR extends whisper.cpp to run the Whisper Encoder on OpenVINO supported devices such as CPU, and Intel GPUs (integrated & discrete).

I've tested this on number of platforms, including

• NUC Intel(R) Core(TM) i7-6770HQ ('Skylake' Skull Canyon NUC) running Ubuntu 22.04
• Core(TM) i7-1185G7 ('Tiger Lake' laptop) running Windows 11 Pro
• Core(TM) i7-12700 ('Alder Lake' Beast Canyon NUC) with installed Intel(R) ARC(TM) A770 discrete graphics card, running Windows 11 Pro

For each platform, the performance of using OpenVINO-based encoder gives a great boost in performance over the default encoder -- even for CPU -- and the ability to easily offload to another OpenVINO-supported device by simply specifying a different string at runtime (e.g. "CPU" --> "GPU") is very convenient.

High-level description of changes

This introduction of OpenVINO Encode support is modeled very closely to how whisper.cpp uses CoreML (this should be pretty obvious in the change-set). If the project is built with OpenVINO support, an OpenVINO-specific encoder is pulled into the build and instantiated at application startup time.

Also similar to CoreML, the models required to be present to take advantage of the OpenVINO encoder can be generated using a new python script in 'models' directory.

Just to point out -- something that does differ between CoreML and the new OpenVINO integration is how/when support is enabled at runtime. CoreML is enabled within the call to whisper_init_*. For OpenVINO, because we want the ability to specify a device string ("CPU", "GPU", etc.), I exposed a new API that is dedicated to initializing OpenVINO, given a ctx:

(in whisper.h):

    // Given a context, enable use of OpenVINO for encode inference. 
    // openvino_model_path: Optional path to OpenVINO encoder IR model. If set to nullptr,
    //                      the path will be generated from the ggml model path that was passed
    //                      in to whisper_init_from_file. For example, if 'path_model' was
    //                      "/path/to/ggml-base.en.bin", then OpenVINO IR model path will be 
    //                      assumed to be "/path/to/ggml-base.en-encoder-openvino.xml".
    // openvino_device: OpenVINO device to run inference on ("CPU", "GPU", etc.)
    // openvino_cache_dir: Optional cache directory that can speed up init time, especially for 
    //                     GPU, by caching compiled 'blobs' there.
    //                     Set to nullptr if not used.
    // Returns 1 on success. If OpenVINO is not enabled in build, this
    // simply returns 0.
    WHISPER_API int whisper_ctx_init_openvino_encoder(struct whisper_context* ctx,
        const char* openvino_model_path,
        const char* openvino_device,
        const char* openvino_cache_dir);

I'm happy to rework this if anyone has a better idea of how to enable OpenVINO support at init time.

main.cpp exposes a new parameter for user to set OpenVINO encode inference device (default is "CPU"):

...
else if (arg == "-oved" || arg == "--ov-e-device")    { params.openvino_encode_device = argv[++i]; }
...

And the new whisper_ctx_init_openvino_encoder API is called right after ctx creation:

   // whisper init

    struct whisper_context * ctx = whisper_init_from_file(params.model.c_str());

    if (ctx == nullptr) {
        fprintf(stderr, "error: failed to initialize whisper context\n");
        return 3;
    }

    // initialize openvino encoder. This has no effect on whisper.cpp builds that don't have OpenVINO configured.
    whisper_ctx_init_openvino_encoder(ctx, nullptr, params.openvino_encode_device.c_str(), nullptr);

How to generate models and enable OpenVINO for whisper.cpp builds

Here are the instructions for generating the OpenVINO models for use with OpenVINO-enabled builds of whisper.cpp:

• First, setup python virtual env. and install python dependencies. Python 3.10 is recommended.

  Windows:

  cd models
  python -m venv openvino_conv_env
  openvino_conv_env\Scripts\activate
  python -m pip install --upgrade pip
  pip install -r openvino-conversion-requirements.txt
  

  Linux and macOS:

  cd models
  python3 -m venv openvino_conv_env
  source openvino_conv_env/bin/activate
  python -m pip install --upgrade pip
  pip install -r openvino-conversion-requirements.txt
  

• Generate an OpenVINO encoder model. For example, to generate a base.en model, use:

  python convert-whisper-to-openvino.py --model base.en
  

  This will produce ggml-base.en-encoder-openvino.xml/.bin IR model files. It's recommended to relocate these to the same folder as ggml models, as that is the default location that the OpenVINO extension will search at runtime.

• Build whisper.cpp with OpenVINO support:

  Download OpenVINO package from release page. The recommended version to use is 2023.0.0.

  After downloading & extracting package onto your development system, set up required environment by sourcing setupvars script. For example:

  Linux:

  source /path/to/l_openvino_toolkit_ubuntu22_2023.0.0.10926.b4452d56304_x86_64/setupvars.sh

  Windows (cmd):

  C:\Path\To\w_openvino_toolkit_windows_2023.0.0.10926.b4452d56304_x86_64\setupvars.bat
  

  And then build the project using cmake:

  cd build
  cmake -DWHISPER_OPENVINO=1 ..

• Run the examples as usual. For example:

  ./main -m models/ggml-base.en.bin -f samples/jfk.wav
  
  ...
  
  whisper_ctx_init_openvino_encoder: loading OpenVINO model from 'models/ggml-base.en-encoder-openvino.xml'
  whisper_ctx_init_openvino_encoder: first run on a device may take a while ...
  whisper_openvino_init: path_model = models/ggml-base.en-encoder-openvino.xml, device = CPU, cache_dir = models/ggml-base.en-encoder-openvino-cache
  whisper_ctx_init_openvino_encoder: OpenVINO model loaded
  
  system_info: n_threads = 4 / 8 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | COREML = 0 | OPENVINO = 1 |
  
  ...

  The first time run on an OpenVINO device is slow, since the OpenVINO framework will compile the IR (Intermediate Representation) model to a device-specific 'blob'. This device-specific blob will get
  cached for the next run.

You can use -oved [DEVICE] argument to main to specify OpenVINO device to offload encoder inference to. For example:

main -m ggml-base.bin -f gb1.wav -oved GPU