Use clap to achieve prompt controlled generation (#223)
* 快速分类音频并把yml格式结果存在训练根目录里 (#190) * Add files via upload * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Update models.py * Update webui.py * Update infer.py * Create compress_model.py * 重新提交，更新Gradio推理UI (#193) * Update webui.py * Update webui.py * 更新 train_ms.py * 更新 models.py * 更新 models.py * 更新 models.py * 更新 train_ms.py * 更新 train_ms.py * 更新 models.py * Update preprocess_text.py * Update config.json * Update train_ms.py * Update webui.py (#206) * Add files via upload (#209) * Update train_ms.py * Update train_ms.py * Update preprocess_text.py * Update train_ms.py * fix (#211) * Update emotion_clustering.py * Add files via upload * Update emotion_clustering.py * add cluster center save * Add files via upload * Update config.py * Update default_config.yml * Update config.py * Update config.py * Update emotion_clustering.py * Update emotion_clustering.py * Update config.py * Update emotion_clustering.py * Update emotion_clustering.py * Update webui.py * Update emotion_clustering.py * Update commons.py * Update emotion_clustering.py * Update webui.py * Update webui.py * Add files via upload * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update train_ms.py * Update webui.py * Update emotion_clustering.py * Update emotion_clustering.py * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix default_config.yml. * Update infer.py * feat: support infer 2.1 models * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: support infer 2.1 models 兼容bug修复 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update train_ms.py * Add CLAP * Fix data loader * Fix infer.py * Fix webui.py * Add prompt template * Update clap_gen.py * Fix wrong environ value * Add g for dur disc * Update clap_gen.py * Fix multilang generation * Update config.json * Prompt mode * Improve slice segments performance * Add preprocess webui * Update webui_preprocess.py * Update webui_preprocess.py * Update config.py * Update default_config.yml * Update config.py * Update clap_gen.py * Delete emo_gen.py * Delete get_emo.py * Delete emotional/wav2vec2-large-robust-12-ft-emotion-msp-dim directory * Update README.md * Update README * Split val per lang * Delete emotion_clustering.py * Update default_config.yml * Update default_config.yml * Update config.py * Update preprocess_text.py * Update webui_preprocess.py * Update defalut_config.yml * Update webui_preprocess.py * Update preprocess_text.py * Random augmentation for CLAP * Update data_utils.py * Update preprocess_text.py * Add vq for CLAP features to avoid overfitting * Random dummy inputs * Update webui.py * Update models.py * Update infer.py * Apply Code Formatter Change * Update config.json * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: YYuX-1145 <138500330+YYuX-1145@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sora <654163754@qq.com> Co-authored-by: Sihan Wang <wangsihan1995@gmail.com> Co-authored-by: Stardust-minus <Stardust-minus@users.noreply.github.com>
This commit is contained in:
Stardust·减
34
emotional/clap-htsat-fused/.gitattributes
vendored
Normal file
34
emotional/clap-htsat-fused/.gitattributes
vendored
Normal file
@@ -0,0 +1,34 @@
|
||||
*.7z filter=lfs diff=lfs merge=lfs -text
|
||||
*.arrow filter=lfs diff=lfs merge=lfs -text
|
||||
*.bin filter=lfs diff=lfs merge=lfs -text
|
||||
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
||||
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
||||
*.ftz filter=lfs diff=lfs merge=lfs -text
|
||||
*.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.h5 filter=lfs diff=lfs merge=lfs -text
|
||||
*.joblib filter=lfs diff=lfs merge=lfs -text
|
||||
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
*.model filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
*.npy filter=lfs diff=lfs merge=lfs -text
|
||||
*.npz filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.parquet filter=lfs diff=lfs merge=lfs -text
|
||||
*.pb filter=lfs diff=lfs merge=lfs -text
|
||||
*.pickle filter=lfs diff=lfs merge=lfs -text
|
||||
*.pkl filter=lfs diff=lfs merge=lfs -text
|
||||
*.pt filter=lfs diff=lfs merge=lfs -text
|
||||
*.pth filter=lfs diff=lfs merge=lfs -text
|
||||
*.rar filter=lfs diff=lfs merge=lfs -text
|
||||
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
||||
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
||||
*.tflite filter=lfs diff=lfs merge=lfs -text
|
||||
*.tgz filter=lfs diff=lfs merge=lfs -text
|
||||
*.wasm filter=lfs diff=lfs merge=lfs -text
|
||||
*.xz filter=lfs diff=lfs merge=lfs -text
|
||||
*.zip filter=lfs diff=lfs merge=lfs -text
|
||||
*.zst filter=lfs diff=lfs merge=lfs -text
|
||||
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
||||
107
emotional/clap-htsat-fused/README.md
Normal file
107
emotional/clap-htsat-fused/README.md
Normal file
@@ -0,0 +1,107 @@
|
||||
---
|
||||
license: apache-2.0
|
||||
---
|
||||
# Model card for CLAP
|
||||
|
||||
Model card for CLAP: Contrastive Language-Audio Pretraining
|
||||
|
||||
|
||||
|
||||
|
||||
# Table of Contents
|
||||
|
||||
0. [TL;DR](#TL;DR)
|
||||
1. [Model Details](#model-details)
|
||||
2. [Usage](#usage)
|
||||
3. [Uses](#uses)
|
||||
4. [Citation](#citation)
|
||||
|
||||
# TL;DR
|
||||
|
||||
The abstract of the paper states that:
|
||||
|
||||
> Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
|
||||
|
||||
|
||||
# Usage
|
||||
|
||||
You can use this model for zero shot audio classification or extracting audio and/or textual features.
|
||||
|
||||
# Uses
|
||||
|
||||
## Perform zero-shot audio classification
|
||||
|
||||
### Using `pipeline`
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from transformers import pipeline
|
||||
|
||||
dataset = load_dataset("ashraq/esc50")
|
||||
audio = dataset["train"]["audio"][-1]["array"]
|
||||
|
||||
audio_classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-fused")
|
||||
output = audio_classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
|
||||
print(output)
|
||||
>>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]
|
||||
```
|
||||
|
||||
## Run the model:
|
||||
|
||||
You can also get the audio and text embeddings using `ClapModel`
|
||||
|
||||
### Run the model on CPU:
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from transformers import ClapModel, ClapProcessor
|
||||
|
||||
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
|
||||
audio_sample = librispeech_dummy[0]
|
||||
|
||||
model = ClapModel.from_pretrained("laion/clap-htsat-fused")
|
||||
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
|
||||
|
||||
inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")
|
||||
audio_embed = model.get_audio_features(**inputs)
|
||||
```
|
||||
|
||||
### Run the model on GPU:
|
||||
|
||||
```python
|
||||
from datasets import load_dataset
|
||||
from transformers import ClapModel, ClapProcessor
|
||||
|
||||
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
|
||||
audio_sample = librispeech_dummy[0]
|
||||
|
||||
model = ClapModel.from_pretrained("laion/clap-htsat-fused").to(0)
|
||||
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")
|
||||
|
||||
inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)
|
||||
audio_embed = model.get_audio_features(**inputs)
|
||||
```
|
||||
|
||||
|
||||
# Citation
|
||||
|
||||
If you are using this model for your work, please consider citing the original paper:
|
||||
```
|
||||
@misc{https://doi.org/10.48550/arxiv.2211.06687,
|
||||
doi = {10.48550/ARXIV.2211.06687},
|
||||
|
||||
url = {https://arxiv.org/abs/2211.06687},
|
||||
|
||||
author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
|
||||
|
||||
keywords = {Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},
|
||||
|
||||
title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
|
||||
|
||||
publisher = {arXiv},
|
||||
|
||||
year = {2022},
|
||||
|
||||
copyright = {Creative Commons Attribution 4.0 International}
|
||||
}
|
||||
```
|
||||
207
emotional/clap-htsat-fused/config.json
Normal file
207
emotional/clap-htsat-fused/config.json
Normal file
@@ -0,0 +1,207 @@
|
||||
{
|
||||
"_commit_hash": null,
|
||||
"architectures": [
|
||||
"ClapModel"
|
||||
],
|
||||
"audio_config": {
|
||||
"_name_or_path": "",
|
||||
"add_cross_attention": false,
|
||||
"aff_block_r": 4,
|
||||
"architectures": null,
|
||||
"attention_probs_dropout_prob": 0.0,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": null,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"cross_attention_hidden_size": null,
|
||||
"decoder_start_token_id": null,
|
||||
"depths": [
|
||||
2,
|
||||
2,
|
||||
6,
|
||||
2
|
||||
],
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"drop_path_rate": 0.0,
|
||||
"early_stopping": false,
|
||||
"enable_fusion": true,
|
||||
"enable_patch_fusion": true,
|
||||
"enable_patch_layer_norm": true,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": null,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"flatten_patch_embeds": true,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"fusion_num_hidden_layers": 2,
|
||||
"fusion_type": null,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 768,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"initializer_factor": 1.0,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"layer_norm_eps": 1e-05,
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"min_length": 0,
|
||||
"mlp_ratio": 4.0,
|
||||
"model_type": "clap_audio_model",
|
||||
"no_repeat_ngram_size": 0,
|
||||
"num_attention_heads": [
|
||||
4,
|
||||
8,
|
||||
16,
|
||||
32
|
||||
],
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_classes": 527,
|
||||
"num_hidden_layers": 4,
|
||||
"num_mel_bins": 64,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": null,
|
||||
"patch_embed_input_channels": 1,
|
||||
"patch_embeds_hidden_size": 96,
|
||||
"patch_size": 4,
|
||||
"patch_stride": [
|
||||
4,
|
||||
4
|
||||
],
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"projection_dim": 512,
|
||||
"projection_hidden_act": "relu",
|
||||
"projection_hidden_size": 768,
|
||||
"pruned_heads": {},
|
||||
"qkv_bias": true,
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"sep_token_id": null,
|
||||
"spec_size": 256,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": true,
|
||||
"tokenizer_class": null,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torch_dtype": null,
|
||||
"torchscript": false,
|
||||
"transformers_version": "4.27.0.dev0",
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false,
|
||||
"window_size": 8
|
||||
},
|
||||
"hidden_size": 768,
|
||||
"initializer_factor": 1.0,
|
||||
"logit_scale_init_value": 14.285714285714285,
|
||||
"model_type": "clap",
|
||||
"num_hidden_layers": 16,
|
||||
"projection_dim": 512,
|
||||
"projection_hidden_act": "relu",
|
||||
"text_config": {
|
||||
"_name_or_path": "",
|
||||
"add_cross_attention": false,
|
||||
"architectures": null,
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"bad_words_ids": null,
|
||||
"begin_suppress_tokens": null,
|
||||
"bos_token_id": 0,
|
||||
"chunk_size_feed_forward": 0,
|
||||
"classifier_dropout": null,
|
||||
"cross_attention_hidden_size": null,
|
||||
"decoder_start_token_id": null,
|
||||
"diversity_penalty": 0.0,
|
||||
"do_sample": false,
|
||||
"early_stopping": false,
|
||||
"encoder_no_repeat_ngram_size": 0,
|
||||
"eos_token_id": 2,
|
||||
"exponential_decay_length_penalty": null,
|
||||
"finetuning_task": null,
|
||||
"forced_bos_token_id": null,
|
||||
"forced_eos_token_id": null,
|
||||
"fusion_hidden_size": 768,
|
||||
"fusion_num_hidden_layers": 2,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 768,
|
||||
"id2label": {
|
||||
"0": "LABEL_0",
|
||||
"1": "LABEL_1"
|
||||
},
|
||||
"initializer_factor": 1.0,
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 3072,
|
||||
"is_decoder": false,
|
||||
"is_encoder_decoder": false,
|
||||
"label2id": {
|
||||
"LABEL_0": 0,
|
||||
"LABEL_1": 1
|
||||
},
|
||||
"layer_norm_eps": 1e-12,
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"max_position_embeddings": 514,
|
||||
"min_length": 0,
|
||||
"model_type": "clap_text_model",
|
||||
"no_repeat_ngram_size": 0,
|
||||
"num_attention_heads": 12,
|
||||
"num_beam_groups": 1,
|
||||
"num_beams": 1,
|
||||
"num_hidden_layers": 12,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_scores": false,
|
||||
"pad_token_id": 1,
|
||||
"position_embedding_type": "absolute",
|
||||
"prefix": null,
|
||||
"problem_type": null,
|
||||
"projection_dim": 512,
|
||||
"projection_hidden_act": "relu",
|
||||
"projection_hidden_size": 768,
|
||||
"pruned_heads": {},
|
||||
"remove_invalid_values": false,
|
||||
"repetition_penalty": 1.0,
|
||||
"return_dict": true,
|
||||
"return_dict_in_generate": false,
|
||||
"sep_token_id": null,
|
||||
"suppress_tokens": null,
|
||||
"task_specific_params": null,
|
||||
"temperature": 1.0,
|
||||
"tf_legacy_loss": false,
|
||||
"tie_encoder_decoder": false,
|
||||
"tie_word_embeddings": true,
|
||||
"tokenizer_class": null,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torch_dtype": null,
|
||||
"torchscript": false,
|
||||
"transformers_version": "4.27.0.dev0",
|
||||
"type_vocab_size": 1,
|
||||
"typical_p": 1.0,
|
||||
"use_bfloat16": false,
|
||||
"use_cache": true,
|
||||
"vocab_size": 50265
|
||||
},
|
||||
"torch_dtype": "float32",
|
||||
"transformers_version": null
|
||||
}
|
||||
50001
emotional/clap-htsat-fused/merges.txt
Normal file
50001
emotional/clap-htsat-fused/merges.txt
Normal file
File diff suppressed because it is too large
Load Diff
22
emotional/clap-htsat-fused/preprocessor_config.json
Normal file
22
emotional/clap-htsat-fused/preprocessor_config.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"chunk_length_s": 10,
|
||||
"feature_extractor_type": "ClapFeatureExtractor",
|
||||
"feature_size": 64,
|
||||
"fft_window_size": 1024,
|
||||
"frequency_max": 14000,
|
||||
"frequency_min": 50,
|
||||
"hop_length": 480,
|
||||
"max_length_s": 10,
|
||||
"n_fft": 1024,
|
||||
"nb_frequency_bins": 513,
|
||||
"nb_max_frames": 1000,
|
||||
"nb_max_samples": 480000,
|
||||
"padding": "repeatpad",
|
||||
"padding_side": "right",
|
||||
"padding_value": 0.0,
|
||||
"processor_class": "ClapProcessor",
|
||||
"return_attention_mask": false,
|
||||
"sampling_rate": 48000,
|
||||
"top_db": null,
|
||||
"truncation": "fusion"
|
||||
}
|
||||
15
emotional/clap-htsat-fused/special_tokens_map.json
Normal file
15
emotional/clap-htsat-fused/special_tokens_map.json
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"bos_token": "<s>",
|
||||
"cls_token": "<s>",
|
||||
"eos_token": "</s>",
|
||||
"mask_token": {
|
||||
"content": "<mask>",
|
||||
"lstrip": true,
|
||||
"normalized": false,
|
||||
"rstrip": false,
|
||||
"single_word": false
|
||||
},
|
||||
"pad_token": "<pad>",
|
||||
"sep_token": "</s>",
|
||||
"unk_token": "<unk>"
|
||||
}
|
||||
100362
emotional/clap-htsat-fused/tokenizer.json
Normal file
100362
emotional/clap-htsat-fused/tokenizer.json
Normal file
File diff suppressed because it is too large
Load Diff
16
emotional/clap-htsat-fused/tokenizer_config.json
Normal file
16
emotional/clap-htsat-fused/tokenizer_config.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"add_prefix_space": false,
|
||||
"bos_token": "<s>",
|
||||
"cls_token": "<s>",
|
||||
"eos_token": "</s>",
|
||||
"errors": "replace",
|
||||
"mask_token": "<mask>",
|
||||
"model_max_length": 512,
|
||||
"pad_token": "<pad>",
|
||||
"processor_class": "ClapProcessor",
|
||||
"sep_token": "</s>",
|
||||
"special_tokens_map_file": null,
|
||||
"tokenizer_class": "RobertaTokenizer",
|
||||
"trim_offsets": true,
|
||||
"unk_token": "<unk>"
|
||||
}
|
||||
1
emotional/clap-htsat-fused/vocab.json
Normal file
1
emotional/clap-htsat-fused/vocab.json
Normal file
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user