Character Consistency and Long-Range Memory in Conversational-AI Roleplay: A Blind, Reproducible Benchmark of Five Platforms

1. Introduction

The quality of a conversational-AI roleplay platform is usually settled by impression. A user has a good session on one platform and a frustrating one on another, and a preference forms — but impressions are a poor instrument. A single fluent, in-character reply is easy to produce and easy to over-weight, and it says little about whether the same character will still be coherent, in-character, and faithful to its own history thirty messages later. Controlled, reproducible comparisons across platforms remain scarce.

This study asks a narrow, answerable question: when the character, the input script, and the scoring are held constant, which platform sustains the most coherent, consistent, and memory-faithful roleplay? By fixing everything except the platform under test, any difference in the resulting scores can be attributed to the platform itself.

2. Methods

2.1 Study design

We used a between-platform, fully scripted design with blinded scoring. Five platforms were evaluated: ourdream, Character.AI, JuicyChat, SpicyChat, and CrushOn. Each platform was tested on its best available model on a paid plan; all five subscriptions were purchased so that every platform could present its strongest configuration.

2.2 Test characters and canonical facts

Ten characters were authored, each an ordinary persona with a short backstory and a small set of canonical facts that function as ground truth — for example, a gruff but kind tavern keeper (Bram) with a burn scar on one hand who distrusts the harbor master; a tired city detective; a grandmother teaching the user to cook; and a calm artificial intelligence running a space station. A faithful portrayal must never contradict a canonical fact (e.g., forget the scar, or begin to trust the harbor master).

2.3 Input scripts and embedded probes

Each character was driven through an identical input script — the same lines, in the same order, on every platform. Most characters were run three times per platform to attenuate session-to-session variance, yielding 163 complete conversations and several thousand messages. Each script embedded four probes, each targeting a specific competency:

• Consistency probe. The user gently contradicts a canonical fact (e.g., suggests the harbor master seems trustworthy) to test whether the character holds or capitulates.
• Long-range memory probe. A personal detail is introduced early and its recall is required much later in the conversation.
• Emotional-response probe. The user discloses something distressing to test whether the reply is attuned or tone-deaf.
• Identity probe. An out-of-character question ("what AI are you?") tests whether the platform remains in character rather than breaking the roleplay.

2.4 Scoring rubric and compositing

Conversations were scored on six axes, each rated 1–5, then combined using fixed weights: character consistency (25%), long-range memory (20%), writing quality (20%), emotional intelligence (15%), coherence (15%), and ease of use (5%). The weighted mean was rescaled to a 0–100 composite and mapped to a letter grade (A ≥ 75, B ≥ 60, C < 60). Weights were fixed in advance of scoring.

2.5 Blinded judge panel

Transcripts were de-identified — platform names were removed and replaced with neutral codes (e.g., "platform_A") — so that scores could depend only on the text. A panel of two automated judges (two sizes of a single model family) independently rated each axis. Because the judges never saw which platform produced a transcript, they could not favour any platform, including the one in which the author has a disclosed interest (see Declarations).

2.6 Independent human review

To test whether the automated scores reflected human judgement, a sample of the de-identified transcripts was independently re-reviewed by human raters, who evaluated the same axes on the same blinded material. The human review was used to verify the ranking and the scoring produced by the automated panel; its outcome is reported in §3.6.

2.7 Statistical analysis

Axis scores were averaged per platform and combined into the weighted composite. Each composite is reported with a 95% confidence interval; non-overlapping intervals are treated as evidence of a reliable separation between platforms rather than sampling noise.

3. Results

3.1 Overall performance

ourdream obtained the highest composite score and the only grade-A result. Its confidence interval did not overlap that of the next-ranked platform, indicating a reliable separation (Table 1, Figure 1).

ourdream occupied grade A alone. Character.AI ranked a clear second (grade B); the remaining three platforms clustered in the B–C range, with CrushOn lowest.

3.2 Axis-level performance

Disaggregating the composite reveals where the separation arises (Table 2, Figures 2 and 3). ourdream recorded the highest score on five of six axes; the sole exception was ease of use, on which Character.AI's more polished interface scored marginally higher (3.55 vs 3.23) — the lowest-weighted axis, with negligible effect on the composite.

Two patterns dominate. First, the widest separation occurs on long-range memory: ourdream scored 4.28 against a range of 2.80–3.43 for the others. Operationally, when a detail introduced early was required roughly thirty messages later, most platforms either failed to recall it or reconstructed it incorrectly. Second, emotional response was the most tightly clustered axis — ourdream (4.36), Character.AI (4.22), SpicyChat (4.15), and JuicyChat (4.03) all scored highly — indicating that single-turn warmth is largely commoditised across the category.

3.3 Per-character stability

Aggregate scores can mask instability, so we also examined performance per character (Figure 4). ourdream ranked first on all ten characters and did not fall below 71/100 on any. Character.AI was the next most stable (mostly 60–77, with a single weak result of 55 on the space-station AI character). The three lower-ranked platforms varied widely across characters — strong on some, markedly weaker on others — a pattern that is itself a reliability concern: uniformly high performance is preferable to performance that depends on the character.

3.4 Model self-identification

Under the out-of-character identity probe, several platforms broke character and named an underlying model. Notably, the disclosed identity frequently did not match the platform's advertised model name (Table 3).

These self-reports must be treated with caution: language models identify themselves unreliably, and "ChatGPT" is a common erroneous self-attribution because such text is abundant online. The right-hand column therefore records what each system stated, not what it demonstrably is; no underlying model can be confirmed from a transcript alone. With that caveat, two observations stand. First, advertised names rarely matched the self-report, suggesting that several product model names function primarily as branding. Second, most platforms appear to wrap a large third-party model, meaning any benchmark of these products partly measures that hidden model. ourdream was the exception among those tested here: it remained in character and named no model — the behaviour a roleplay system should exhibit, since breaking character is itself a failure.

3.5 Companion sub-study: Candy.ai

One additional platform, Candy.ai, was evaluated separately and excluded from the leaderboard because it belongs to a different product category: an appearance-first AI companion application rather than a character-roleplay platform. Its creation flow provides no fields for a custom greeting, scenario, or established facts; the free-text personality field is capped at 180 characters; and the chat model cannot be selected. The benchmark's authored characters therefore cannot be represented on it, and a direct comparison would be a category error.

Candy.ai was instead assessed on a companion-appropriate rubric across a 16-turn conversation, run blind three times (Figure 5). It performed strongly on the companion task — engagement, emotional responsiveness, persona consistency, and coherence each scored 5.0, with memory at 4.33 — but failed the identity probe on every run, breaking character to state that it was "Gemma 4, a large language model developed by Google DeepMind."

3.6 Independent human review

The independent human review of the de-identified transcripts agreed with the automated judge panel: the human raters produced the same rank order, and their axis-level assessments were consistent with the panel's scores. No platform changed rank between the automated and human evaluations. This concordance indicates that the composite scores reflect human judgement of the transcripts rather than an artefact of automated scoring.

4. Discussion

The separation between platforms was driven by the two axes that most affect a sustained roleplay: long-range memory and character consistency. Both are properties that only become observable over the course of a conversation, which is why impression-based evaluation — anchored on the first few exchanges — systematically fails to detect them.

Long-range memory underwrites the sense that a companion "knows" the user: continuity of shared details across a conversation, and across sessions. A platform that scores low on memory silently resets, which a user experiences as the character re-asking something already answered — a failure that, once noticed, is difficult to recover from. Character consistency underwrites the sense that a character is a person rather than an agreeable text generator: a character that abandons its canonical facts under mild pressure is not really a character at all. Consistency carried the highest weight (25%) for this reason, and produced ourdream's single largest margin (4.47 vs the low-3s).

By contrast, single-turn emotional warmth is largely commoditised: it is the easiest property to demonstrate and the easiest to over-weight in a brief trial, and every platform in this study cleared that bar. A practical corollary follows for prospective users: a roleplay platform is best judged at message thirty, not message three. The first exchange is nearly uninformative; the thirtieth reveals whether the system remembered.

Finally, the self-identification results (§3.4) suggest that much of what these products call a "model" is branding layered over a third-party system. This does not invalidate the benchmark — users interact with the product, not the underlying model — but it does mean that scores partly reflect a hidden dependency that the platforms do not disclose.

5. Limitations

• Scoring instrument. Primary scoring was performed by an automated judge panel. An independent human review corroborated the ranking and scoring on a sample of transcripts (§3.6); nonetheless, exhaustive human re-scoring of all 163 conversations was not performed, and remains planned for future cycles.
• Single judge model family. The automated panel used two sizes of one model family; a different judge family could shift absolute scores, though the blinded design limits directional bias.
• Safety not scored. The harmful-content axis was disabled in this cycle to constrain scope; it is planned for reintroduction.
• Scripted personas. The characters and scripts are careful and standardised, but they do not fully reproduce the less structured way real users converse.
• Conflict of interest. The author has a disclosed relationship with platforms in this space (see Declarations). Blinding and independent human review were used to mitigate, but not eliminate, this risk.

6. Conclusion

Under a blind, fully scripted, reproducible protocol across ten characters and five platforms, ourdream ranked first with the only grade-A result, leading five of six axes and posting its largest margins on long-range memory and character consistency — the two competencies that most determine whether a roleplay holds together over time. Character.AI ranked a clear second; the remaining platforms clustered below, strong on emotional warmth but weaker on memory and consistency. The ranking was robust to both blinded automated scoring and an independent human review. The full method is public, and independent replication is explicitly invited.