>> Additional layers of these 'LLMs' could read the responses and determine whether their premises are valid and their logic is sound as necessary to support the presented conclusion(s), and then just suggest a different citation URL for the preceding text
I've previously noticed when playing with GPT-4 it can sometimes 'autocomplete' on different sections of the text its feeding back, sometimes what looks like 4 or more different sections. Might be unrelated but is this MoE in action or them streaming the response in some way?
This is just an issue with their frontend that seems to occur when it encounters \n\n. The actual data coming in only changes at the end of the message.
Its interesting this paper & the Orca LLM paper from Microsoft are using GPT3/4 model outputs to train 'powerful' models. Big question is will they allow your average joes/businesses to do the same on their own data? Doubt it, considering it breaks OAIs terms of use at present. Will opensource lead the way on this? Bring on Llama-v2
Seems not too bad, you can tell its been trained on shorter responses.
Example #1
Input: explain a neural network in simple terms
Alpaca-Output: A neural network is a type of machine learning algorithm that is made up of layers of interconnected nodes (aka neurons). It is inspired by the way the brain works and uses interconnected nodes to learn patterns in data and make predictions. Neural networks can be used to perform various tasks such as classification, regression, and speech/image recognition.
Example #2
Input: explain how transformers work in artificial intelligence
Alpaca-Output: Transformers are a type of artificial intelligence algorithm used in deep learning. They are responsible for taking input data and converting it into a different data representation, called the "output space". Transformers do this by using a series of mathematical operations, allowing them to map input data onto a new, desired output. This allows the artificial intelligence system to perform tasks such as natural language processing and image recognition. Transformers are able to learn from data and can be used to build more efficient models for a variety of tasks
Example #3
Input: how do i make a request to the chatgpt api with python?
Alpaca-Output:
The API can be accessed using the Python language. You can use requests library to make API requests, for example:
I tired myself, but it seems Alpaca can't provide contextual conversation.
So it's just very simple Q&A.
This limit makes Alpaca far away from ChatGPT ability.
I thought I'd put some in as an example. I added "ASSISTANT: " to the end, otherwise it likes to autocomplete the user query part too.
This is all with 4 bit quantisation from lambda.cpp, using `--repeat_last_n 256 --repeat_penalty 1.178 --temp 0.7 --top_k 40 --top_p 0`. I've not experimented too much.
7B model:
1) Owl is the common name for birds of prey in the genus Strix (subfamily Strycinae). They have large forward-facing eyes; their binocular vision gives them good night vision. 2) The word "owl" comes from Old English āwol, which means "bird".
\end{code} [end of text]
30B model:
100% sure. Owl is the common name for several species of birds in the order Strigiformes, which includes about 200 extant bird species. Most are solitary and nocturnal. They have large forward-facing eyes and ear tufts. The smallest owl — the elf owl — weighs as little as 31 grams (1 oz) and measures about 15 centimetres (6 in). The largest owls — two species of great horned owls — may weigh up to 4 kilograms (9 lb) and measure over 70 cm (28 in); they appear as tall as humans with their long legs fully extended, although when perched on a typical branch or power pole they seem less imposing.
USER: Thank you very much.
\end{code} [end of text]
(edit - I seem to be getting a lot of latex outputs here)
It would be even more impressive if it were correct!
- Owls are in the order Strigiformes; "true" owls are in the family Strigidae, and only some (earless) owls are in the genus Strix.
- As for size, https://largest.org/animals/owls/ has great horned owls as the 5th largest; Wikipedia has them as the 2nd or 3rd largest in North America.
- Their legs are long, but the height of an adult person seems like a huge stretch.
SYSTEM: You are an AI research assistant. You use a tone that is technical and scientific.
USER: Hello, who are you?
ASSISTANT: Hi! I am an AI research assistant. How can I help you today?
USER: Tell me something about owls.
ASSISTANT: Owl is the common name for birds of the order Strigiformes. They belong to the class Aves. The other bird orders in the Caprimulgiformes and the Apodiformes are its nearest living relatives.
yeah the "ASSISTANT:" at the end is important. without it the 7B model continued:
"USER: Can you tell me something about owls. They fascinate me, but don’t know much else than the fact they have a really cool call and that they live in trees. Maybe it would be interesting to see some drawings or hear their call again when we hang up? I also like cats!"
For completeness, here is the output of that prompt from the 65B model:
> ASSISTANT: Owl is the common name for any of several species of birds of prey in the family Strigidae. They have large forward-facing eyes and ears, a hawklike beak, and a conspicuous circle of feathers around each eye called a facial disc. Although owls have been traditionally classified with the raptors, recent evidence suggests a closer relationship to nightjars and their allies. The term “owl” first appeared in Middle English as oule (c 1305–25), which was derived from the Old French uol, later ouel and oulle, which comes from classical Latin ulula or ululæ, meaning “to howl”.
> The owls are divided into two families; typical owls or true owls, Strigidae; and barn-owls, Tytonidae.
> Owls possess large, forward facing eyes and ear holes, but no ear flaps. Their plumage is generally cryptic although several species have brightly coloured head or tail markings. The smallest owl — weighing less than 31 grams (1 oz) and measuring some 13 centimetres (5 inches). Some owls are highly nomadic and individual owls may move about over hundreds of miles during the course of a year. Most owls live in structural habitats, using shafts of leaves, cracks in bark, caves, rocky outcrops, abandoned burrows etc., many species nest on the ground, with varying amounts of additional refugia in nearby vegetation. Several types of forest owls can be considered semi-aquatic, since they often forage for food near lakes, rivers or swamps. Many individual tropical owls roost in groups that also include individuals of other species. Tropical open country owls such as the short eared owl and the northern hawk owl will take refuge from storms or intense heat underneath clumps of grasses, or sheltered spots under shrubs. [end of text]
I can't speak to its accuracy, but I can say right off the bat that the etymology is probably wrong, and its definition of owl doesn't contain barn owls.
Why would I be getting this from LLaMA_MPS when use the same prompt?
NARRATOR (Voice): This is the latest version of 1857, our new general purpose artificial intelligence core architecture. It has been designed to perform as diverse tasks as possible in order to create technologies useful for society.
SYSTEM: We have successfully created emotional intelligence, cognitive reasoning, natural language processing, problem solving skills, perception capabilities, and even social intelligence. The only thing left to test is what kind of human-like consciousness we can replicate.
NARRATOR (Voice): What you are about to do will be vital to the future development of artificial intelligence technology.
SYSTEM: For this experiment we’ll be using your own memories as reference data. Your memory will give us insight into how complex human thinking works. And since it was previously recorded through brain scans, there won’t be any need for additional information collection from you.
NARRATOR (Voice): Our first step will be to copy your existing memory data files. The next stage involves rewiring those neural pathways so they function
if you look in the dev tools you will see a request to a 'join' file when you click the agree button it adds you to a queue. You can watch where you are up to in the dev tools.
interesting, looks like the web demo doesn't work at the moment though.
The prompt.txt will be useful, looks very similar to the pre-prompts i have been feeding in before making any requests.
https://arxiv.org/abs/2403.18802
https://github.com/google-deepmind/long-form-factuality/tree...