Li)ddlmZddlmZddlmZmZddlZddlmZm Z ddl m Z ddl m Z Gdd e jZy) ) annotations)Iterable)AnyLiteralN)Tensornn)SentenceTransformer)cos_simceZdZedddf d fd Z d d dZ d d dZd dZddZe ddZ xZ S)"GlobalOrthogonalRegularizationLoss?meanct|||_||_||_||_|s |s t d|dvrt d|d||_y)a  Global Orthogonal Regularization (GOR) Loss that encourages embeddings to be well-distributed in the embedding space by penalizing high mean similarities and high second moments of similarities across unrelated inputs. The loss consists of two terms: 1. Mean term: Penalizes when the mean similarity across unrelated embeddings is high 2. Second moment term: Penalizes when the second moment of similarities is high A high second moment indicates that some embeddings have very high similarities, suggesting clustering or concentration in certain regions of the embedding space. A low second moment indicates that similarities are more uniformly distributed. The loss is called independently on each input column (e.g., queries and passages) and combines the results using either mean or sum aggregation. This is why the loss can be used on any dataset configuration (e.g., single inputs, pairs, triplets, etc.). It's recommended to combine this loss with a primary loss function, such as :class:`MultipleNegativesRankingLoss`. Args: model: SentenceTransformer model similarity_fct: Function to compute similarity between embeddings (default: cosine similarity) mean_weight: Weight for the mean term loss component. None or 0 can be used to disable this term (default: 1.0) second_moment_weight: Weight for the second moment term loss component. None or 0 can be used to disable this term (default: 1.0) aggregation: How to combine losses across input columns. Either "mean" or "sum" (default: "mean"). The EmbeddingGemma paper uses "sum". References: - For further details, see: https://arxiv.org/abs/1708.06320 or https://arxiv.org/abs/2509.20354. The latter paper uses the equivalent of GOR with ``mean_weight=0.0`` and ``aggregation="sum"``. Inputs: +-------+--------+ | Texts | Labels | +=======+========+ | any | none | +-------+--------+ Example: :: import torch from datasets import Dataset from torch import Tensor from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer from sentence_transformers.losses import GlobalOrthogonalRegularizationLoss, MultipleNegativesRankingLoss from sentence_transformers.util import cos_sim model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) class InfoNCEGORLoss(torch.nn.Module): def __init__(self, model: SentenceTransformer, similarity_fct=cos_sim, scale=20.0) -> None: super().__init__() self.model = model self.info_nce_loss = MultipleNegativesRankingLoss(model, similarity_fct=similarity_fct, scale=scale) self.gor_loss = GlobalOrthogonalRegularizationLoss(model, similarity_fct=similarity_fct) def forward(self, sentence_features: list[dict[str, Tensor]], labels: Tensor | None = None) -> Tensor: embeddings = [self.model(sentence_feature)["sentence_embedding"] for sentence_feature in sentence_features] info_nce_loss: dict[str, Tensor] = { "info_nce": self.info_nce_loss.compute_loss_from_embeddings(embeddings, labels) } gor_loss: dict[str, Tensor] = self.gor_loss.compute_loss_from_embeddings(embeddings, labels) return {**info_nce_loss, **gor_loss} loss = InfoNCEGORLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset=train_dataset, loss=loss, ) trainer.train() Alternatively, you can use multi-task learning to train with both losses: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer from sentence_transformers.losses import GlobalOrthogonalRegularizationLoss, MultipleNegativesRankingLoss from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) mnrl_loss = MultipleNegativesRankingLoss(model) gor_loss = GlobalOrthogonalRegularizationLoss(model) trainer = SentenceTransformerTrainer( model=model, train_dataset={"main": train_dataset, "gor": train_dataset}, loss={"main": mnrl_loss, "gor": gor_loss}, ) trainer.train() zDAt least one of mean_weight or second_moment_weight must be non-zero)rsumz*aggregation must be 'mean' or 'sum', got ''N)super__init__modelsimilarity_fct mean_weightsecond_moment_weight ValueError aggregation)selfrrrrr __class__s /var/www/html/lcp-python-backend/venv/lib/python3.12/site-packages/sentence_transformers/losses/GlobalOrthogonalRegularizationLoss.pyrz+GlobalOrthogonalRegularizationLoss.__init__shX  ,&$8!#7cd d o -I+VWXY Y&cp|Dcgc]}|j|d}}|j|Scc}w)Nsentence_embedding)rcompute_loss_from_embeddings)rsentence_featureslabelssentence_feature embeddingss rforwardz*GlobalOrthogonalRegularizationLoss.forwardsAbssM]djj!123GHs s00<<ts3ct|Dcgc]}|j|c}\}}i}|jrVtj|}|j dk(r|j n|j}|j|z|d<|jrVtj|} |j dk(r| j n| j} |j| z|d<|Scc}w)aZ Compute the GOR loss from pre-computed embeddings. Args: embeddings: List of embedding tensors, one for each input column (e.g., [queries, passages]) labels: Not used, kept for compatibility Returns: Dictionary containing the weighted mean term and second moment term losses rgor_meangor_second_moment) zip compute_gorrtorchstackrrrr) rr$r" embedding mean_termssecond_moment_termsresults stacked_meanaggregated_meanstacked_second_momentaggregated_second_moments rr z?GlobalOrthogonalRegularizationLoss.compute_loss_from_embeddingss+.]g/hPY0@0@0K/h*i' '    ;;z2L484D4D4Ml..0S_SdSdSfO"&"2"2_"DGJ   $ $$)KK0C$D !/3/?/?5/H%))+NcNhNhNj %,0+D+DG_+_G' (0isC1cb|jd}|jd}|j||}|jd||dz z}|j|z j d}|j dj|z }t j |d|z z }||fS)a% Compute the Global Orthogonal Regularization terms for a batch of embeddings. The GOR loss encourages embeddings to be well-distributed by: 1. Mean term (M_1^2): Penalizes high mean similarity, pushing embeddings apart 2. Second moment term (M_2 - 1/d): Penalizes when the second moment exceeds 1/d, encouraging uniform distribution Args: embeddings: Tensor of shape (batch_size, embedding_dim) Returns: Tuple of (mean_term, second_moment_term) losses (unweighted) rgr )sizerfill_diagonal_rpowr+relu) rr$ batch_size hidden_dim sim_matrixnum_off_diagonal mean_term second_momentsecond_moment_terms rr*z.GlobalOrthogonalRegularizationLoss.compute_gors __Q' __Q' ((Z@ !!#&%a8 ^^%(88==a@ #q)--/2BB "ZZ z9I(JK,,,rct|jj|j|j|jdS)N)rrrr)r__name__rrrrs rget_config_dictz2GlobalOrthogonalRegularizationLoss.get_config_dicts6"11::++$($=$=++   rcy)NaO @misc{zhang2017learningspreadoutlocalfeature, title={Learning Spread-out Local Feature Descriptors}, author={Xu Zhang and Felix X. Yu and Sanjiv Kumar and Shih-Fu Chang}, year={2017}, eprint={1708.06320}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/1708.06320}, } rEs rcitationz+GlobalOrthogonalRegularizationLoss.citations r) rr r float | NonerrJrzLiteral['mean', 'sum']returnNone)N)r!zIterable[dict[str, Tensor]]r" Tensor | NonerKdict[str, Tensor])r$z list[Tensor]r"rMrKrN)r$rrKztuple[Tensor, Tensor])rKzdict[str, Any])rKstr) rD __module__ __qualname__r rr%r r*rFpropertyrI __classcell__)rs@rr r s$'-0.4 u'"u'" u' + u' , u' u'pW[=!<=FS= =AE&0= 6-B   rr ) __future__rcollections.abcrtypingrrr+rr)sentence_transformers.SentenceTransformerr sentence_transformers.utilr Moduler rHrrrZs,"$ I.NNr