Views
No views yet
t5-11b. The parameters are unchanged, meaning that the model should be trained to obtain best performance.While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs – which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure – but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM’s architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.
1from transformers import AutoTokenizer, AutoModel
2
3modelcard = 'plenz/GLM-t5-11b'
4
5print('Load the model and tokenizer')
6model = AutoModel.from_pretrained(modelcard, trust_remote_code=True, revision='main')
7tokenizer = AutoTokenizer.from_pretrained(modelcard)
8
9print('get dummy input (2 instances to show batching)')
10graph_1 = [
11 ('black poodle', 'is a', 'dog'),
12 ('dog', 'is a', 'animal'),
13 ('cat', 'is a', 'animal')
14]
15text_1 = 'The dog chased the cat.'
16
17graph_2 = [
18 ('dog', 'is a', 'animal'),
19 ('dog', 'has', 'tail'),
20 ('dog', 'has', 'fur'),
21 ('fish', 'is a', 'animal'),
22 ('fish', 'has', 'scales')
23]
24text_2 = None # only graph for this instance
25
26print('prepare model inputs')
27how = 'global' # can be 'global' or 'local', depending on whether the local or global GLM should be used. See paper for more details.
28data_1 = model.data_processor.encode_graph(tokenizer=tokenizer, g=graph_1, text=text_1, how=how)
29data_2 = model.data_processor.encode_graph(tokenizer=tokenizer, g=graph_2, text=text_2, how=how)
30datas = [data_1, data_2]
31model_inputs = model.data_processor.to_batch(data_instances=datas, tokenizer=tokenizer, max_seq_len=None, device='cpu')
32
33print('compute token encodings')
34outputs = model(**model_inputs)
35
36# get token embeddings
37print('Sequence of tokens (batch_size, max_seq_len, embedding_dim):', outputs.last_hidden_state.shape) # embeddings of all graph and text tokens. Nodes in the graph (e.g., dog) appear only once in the sequence.
38print('embedding of `black poodle` in the first instance. Shape is (seq_len, embedding_dim):', model.data_processor.get_embedding(sequence_embedding=outputs.last_hidden_state[0], indices=data_1.indices, concept='black poodle', embedding_aggregation='seq').shape) # embedding_aggregation can be 'seq' or 'mean'. 'seq' returns the sequence of embeddings (e.g., all tokens of `black poodle`), 'mean' returns the mean of the embeddings.1from transformers import AutoTokenizer, AutoModel, T5ForConditionalGeneration
2
3modelcard = 'plenz/GLM-t5-11b'
4modelcard_generation = 't5-11b'
5
6print('load the model and tokenizer')
7model_generation = T5ForConditionalGeneration.from_pretrained(modelcard_generation)
8del model_generation.encoder # we only need the decoder for generation. Deleting the encoder is optional, but saves memory.
9model = AutoModel.from_pretrained(modelcard, trust_remote_code=True, revision='main')
10tokenizer = AutoTokenizer.from_pretrained(modelcard)
11model_generation.shared = model.shared # share embeddings between encoder and decoder. This mimics the T5 architecture.
12
13print('get dummy input (2 instances to show batching)')
14graph_1 = [
15 ('black poodle', 'is a', 'dog'),
16 ('dog', 'is a', 'animal'),
17 ('cat', 'is a', 'animal')
18]
19text_1 = 'summarize: The black poodle chased the cat.' # with T5 prefix
20
21graph_2 = [
22 ('dog', 'is a', 'animal'),
23 ('dog', 'has', 'tail'),
24 ('dog', 'has', 'fur'),
25 ('fish', 'is a', 'animal'),
26 ('fish', 'has', 'scales')
27]
28text_2 = "Dogs have <extra_id_0> and fish have <extra_id_1>. Both are <extra_id_2>." # T5 MLM
29
30print('prepare model inputs')
31how = 'global' # can be 'global' or 'local', depending on whether the local or global GLM should be used. See paper for more details.
32data_1 = model.data_processor.encode_graph(tokenizer=tokenizer, g=graph_1, text=text_1, how=how)
33data_2 = model.data_processor.encode_graph(tokenizer=tokenizer, g=graph_2, text=text_2, how=how)
34datas = [data_1, data_2]
35model_inputs, attention_mask = model.data_processor.to_batch(data_instances=datas, tokenizer=tokenizer, max_seq_len=None, device='cpu', return_attention_mask=True)
36
37print('compute token encodings')
38outputs = model(**model_inputs)
39
40print('generate conditional on encoded graph and text')
41outputs = model_generation.generate(encoder_outputs=outputs, max_new_tokens=10, attention_mask=attention_mask)
42print('generation 1:', tokenizer.decode(outputs[0], skip_special_tokens=True))
43print('generation 2:', tokenizer.decode(outputs[1], skip_special_tokens=False)) model_generation.shared = model.shared after loading the models. For inference this has no effect, since the embeddings are not updated during inference. However, during training / finetuning, the embeddings can become different for the encoder and decoder if they are not shared.1@inproceedings{plenz-frank-2024-graph,
2 title = "Graph Language Models",
3 author = "Plenz, Moritz and Frank, Anette",
4 booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics",
5 year = "2024",
6 address = "Bangkok, Thailand",
7 publisher = "Association for Computational Linguistics",
8}